https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chris Nogradi

04/25/2023, 2:04 PM
Just tried to upgrade to 1.3.1 and got the following error: Exception ignored in: <function GrpcServerProcess.del at 0x00000237CF519F70> Traceback (most recent call last): File "c:\users\1138247\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1288, in del if not self._shutdown and not self._heartbeat: AttributeError: 'GrpcServerProcess' object has no attribute '_shutdown' Known issue?
👍 1
d

daniel

04/25/2023, 2:10 PM
Hi Chris - was there another error before this? I don't think this was the root cause, looks like an issue cleaning up after some other problem
c

Chris Nogradi

04/25/2023, 2:18 PM
I see this but nothing else: Stack Trace: File "c:\users\1138247\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_core\host_representation\grpc_server_registry.py", line 194, in _get_grpc_endpoint server_process = GrpcServerProcess( File "c:\users\1138247\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1217, in init server_process, self.port = _open_server_process_on_dynamic_port( File "c:\users\1138247\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1153, in _open_server_process_on_dynamic_port server_process = open_server_process( File "c:\users\1138247\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1124, in open_server_process wait_for_grpc_server(server_process, client, subprocess_args, timeout=startup_timeout) File "c:\users\1138247\git\cbm+\d4-orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1058, in wait_for_grpc_server raise Exception(
d

daniel

04/25/2023, 2:18 PM
Is there an error message just before that stack trace?
c

Chris Nogradi

04/25/2023, 2:28 PM
here is the whole dump, is the problem simply the warning? 2023-04-25 085822 -0500 - dagit - INFO - Using temporary directory orchestrator\tmpimw24em6 for storage. This will be removed when dagit exits. 2023-04-25 085822 -0500 - dagit - INFO - To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use. 2023-04-25 085847 -0500 - dagster.builtin - INFO - Executing command: dbt --no-use-color --log-format json ls --project-dir orchestrator\orch\assets\dbt\example --profiles-dir \AppData\Local\Temp\tmpoc3unx_6/example --select * --output json 2023-04-25 085854 -0500 - dagster.builtin - INFO - dbt exited with return code 0 2023-04-25 085908 -0500 - dagster - INFO - Started Dagster code server for package orch on port 54246 in process 29968 2023-04-25 085953 -0500 - dagster - INFO - Shutting down Dagster code server for package orch on port 54246 in process 29968 Exception ignored in: <function GrpcServerProcess.del at 0x00000237CF519F70> Traceback (most recent call last): File "orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1288, in del if not self._shutdown and not self._heartbeat: AttributeError: 'GrpcServerProcess' object has no attribute '_shutdown' orchestrator\.venv\lib\site-packages\dagster\_core\workspace\context.py591 UserWarning: Error loading repository location orchException gRPC server exited with return code 0 while starting up with the command: "orchestrator\.venv\scripts\python.exe -m dagster api grpc --lazy-load-user-code --port 54246 --heartbeat --heartbeat-timeout 45 --fixed-server-id d754c185-73c6-4ae4-930c-da4dbbd69eef --log-level info --inject-env-vars-from-instance --instance-ref {"__class__": "InstanceRef", "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name": "LocalComputeLogManager", "config_yaml": "base_dir: orchestrator\\tmpimw24em6\\storage\n", "module_name": "dagster.core.storage.local_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteEventLogStorage", "config_yaml": "base_dir: orchestrator\\tmpimw24em6\\history\\runs\\\n", "module_name": "dagster.core.storage.event_log"}, "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: orchestrator\\tmpimw24em6\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunLauncher", "config_yaml": "{}\n", "module_name": "dagster"}, "run_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteRunStorage", "config_yaml": "base_dir: orchestrator\\tmpimw24em6\\history\\\n", "module_name": "dagster.core.storage.runs"}, "schedule_storage_data": {"__class__": "ConfigurableClassData", "class_name": "SqliteScheduleStorage", "config_yaml": "base_dir: orchestrator\\tmpimw24em6\\schedules\n", "module_name": "dagster.core.storage.schedules"}, "scheduler_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "secrets_loader_data": null, "settings": {"telemetry": {"enabled": false}}, "storage_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterSqliteStorage", "config_yaml": "base_dir: orchestrator\\tmpimw24em6\n", "module_name": "dagster.core.storage.sqlite_storage"}} --location-name orch --package-name orch" Stack Trace: File "orchestrator\.venv\lib\site-packages\dagster\_core\host_representation\grpc_server_registry.py", line 194, in _get_grpc_endpoint server_process = GrpcServerProcess( File "orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1217, in init server_process, self.port = _open_server_process_on_dynamic_port( File "orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1153, in _open_server_process_on_dynamic_port server_process = open_server_process( File "orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1124, in open_server_process wait_for_grpc_server(server_process, client, subprocess_args, timeout=startup_timeout) File "orchestrator\.venv\lib\site-packages\dagster\_grpc\server.py", line 1058, in wait_for_grpc_server raise Exception( warnings.warn( orchestrator\.venv\lib\site-packages\dagster\_core\execution\compute_logs.py48 UserWarning: WARNING: Compute log capture is disabled for the current environment. Set the environment variable
PYTHONLEGACYWINDOWSSTDIO
to enable. warnings.warn(WIN_PY36_COMPUTE_LOG_DISABLED_MSG) 2023-04-25 085953 -0500 - dagit - INFO - Serving dagit on http://127.0.0.1:3000 in process 14132
d

daniel

04/25/2023, 2:30 PM
Yeah, I'm confused by the sequence of logs there, but it looks like the first thing that goes wrong is that your code server is shutting down immediately on startup. Try running this command locally: "_orchestrator\.venv\scripts\python.exe -m dagster api grpc --lazy-load-user-code --port 54246"_ The expected behavior is that it spins up a server that keeps running, but it looks for some reason its returning right away
maybe running it locally will help explain why
c

Chris Nogradi

04/25/2023, 2:48 PM
When I do that it runs without exiting.
d

daniel

04/25/2023, 2:49 PM
strange - any idea what would be different between you running it locally and dagster running the same command in a subprocess? How long does it take to run and say that the server is running?
c

Chris Nogradi

04/25/2023, 2:58 PM
ok so I reverted back to my original version of 1.1.19 and it is still failing the same way. So must be a problem in the environment. Sorry to waste you time.
d

daniel

04/25/2023, 2:58 PM
no problem - that "Exception ignored" is not expected so we'll get that fixed, even if it wasn't the root cause - so thx for reporting nonetheless
c

Chris Nogradi

04/25/2023, 2:59 PM
haha happy to contribute
FYI, issue was proxy server interference. Now it runs after upgrade except for this crash which I would love to see fixed. Maybe I should contribute my own fix: https://github.com/dagster-io/dagster/issues/13512
d

daniel

04/25/2023, 3:18 PM
ah proxy server makes sense!
2 Views