https://dagster.io/ logo
#ask-community
Title
# ask-community
g

Greg Whittier

12/05/2022, 2:51 PM
I was able to work through the tutorials on my home machine OK, but the exact same setup isn't working on my work machine. I create a fresh conda environment "conda create -n dagster python=3.9" and then pip install dagster dagit. (I've also tried dagster from conda-forge.) At home, it works fine, but om my work machine (both windows) starting the code server times out after 45 seconds with the following.
(dagster3) Z:\foo\dagster>dagit -f cereals.py
Using temporary directory Z:\foo\dagster\tmp96vdgvk3 for storage. This will be removed when dagit exits.
To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.
0it [00:00, ?it/s]
0it [00:00, ?it/s]
2022-12-05 09:45:55 -0500 - dagster - INFO - Started Dagster code server for file cereals.py on port 55246 in process 21112
2022-12-05 09:46:40 -0500 - dagster - INFO - Shutting down Dagster code server for file cereals.py on port 55246 in process 21112
C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_core\workspace\context.py:532: UserWarning: Error loading repository location cereals.py:Exception: gRPC server exited with return code 0 while starting up with the command: "C:\Users\greg\Anaconda3\envs\dagster3\python.exe -m dagster api grpc --lazy-load-user-code --port 55246 --heartbeat --heartbeat-timeout 45 --fixed-server-id becc2332-a6a5-403e-bc7e-b3bf94eee62f --log-level INFO --inject-env-vars-from-instance --instance-ref {"class__": "InstanceRef", "compute_logs_data": {"_class_": "ConfigurableClassData", "class_name": "LocalComputeLogManager", "config_yaml": "base_dir: Z:\\foo\\dagster\\tmp96vdgvk3\\storage\n", "module_name": "dagster.core.storage.local_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"_class_": "ConfigurableClassData", "class_name": "SqliteEventLogStorage", "config_yaml": "base_dir: Z:\\foo\\dagster\\tmp96vdgvk3\\history\\runs\\\n", "module_name": "dagster.core.storage.event_log"}, "local_artifact_storage_data": {"_class_": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: Z:\\foo\\dagster\\tmp96vdgvk3\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"_class_": "ConfigurableClassData", "class_name": "DefaultRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"_class_": "ConfigurableClassData", "class_name": "DefaultRunLauncher", "config_yaml": "{}\n", "module_name": "dagster"}, "run_storage_data": {"_class_": "ConfigurableClassData", "class_name": "SqliteRunStorage", "config_yaml": "base_dir: Z:\\foo\\dagster\\tmp96vdgvk3\\history\\\n", "module_name": "dagster.core.storage.runs"}, "schedule_storage_data": {"_class_": "ConfigurableClassData", "class_name": "SqliteScheduleStorage", "config_yaml": "base_dir: Z:\\foo\\dagster\\tmp96vdgvk3\\schedules\n", "module_name": "dagster.core.storage.schedules"}, "scheduler_data": {"_class_": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "secrets_loader_data": null, "settings": {}, "storage_data": {"_class_": "ConfigurableClassData", "class_name": "DagsterSqliteStorage", "config_yaml": "base_dir: Z:\\foo\\dagster\\tmp96vdgvk3\n", "module_name": "dagster.core.storage.sqlite_storage"}} --location-name cereals.py -f cereals.py -d Z:\foo\dagster"
Stack Trace:
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_core\host_representation\grpc_server_registry.py", line 215, in _get_grpc_endpoint
server_process = GrpcServerProcess(
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_grpc\server.py", line 1170, in _init_
self.server_process, self.port = _open_server_process_on_dynamic_port(
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_grpc\server.py", line 1108, in _open_server_process_on_dynamic_port
server_process = open_server_process(
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_grpc\server.py", line 1080, in open_server_process
wait_for_grpc_server(server_process, client, subprocess_args, timeout=startup_timeout)
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_grpc\server.py", line 1015, in wait_for_grpc_server
raise Exception(
warnings.warn(
C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_core\execution\compute_logs.py:48: UserWarning: WARNING: Compute log capture is disabled for the current environment. Set the environment variable PYTHONLEGACYWINDOWSSTDIO to enable.
warnings.warn(WIN_PY36_COMPUTE_LOG_DISABLED_MSG)
2022-12-05 09:46:40 -0500 - dagit - INFO - Serving dagit on <http://127.0.0.1:3000> in process 17176
I wondered if maybe a port was blocked and tried running with "dagster api grpc --port-num 24266" and then "dagit --grpc-port 24266" I can see the code server listening on the port
PS H:\> netstat -ano | select-string 24246
TCP    127.0.0.1:24246        0.0.0.0:0              LISTENING       23732
TCP    [::1]:24246            [::]:0                 LISTENING       23732
But I can an error from dagit that it can't find the rpc server.
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1670264256.313000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3261,"referenced_errors":[{"created":"@1670264256.313000000","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>
Stack Trace:
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_grpc\client.py", line 139, in _query
return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\dagster\_grpc\client.py", line 129, in _get_response
return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\grpc\_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "C:\Users\greg\Anaconda3\envs\dagster3\lib\site-packages\grpc\_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
dagster bot responded by community 1
OK, this drove me crazy. It seems to be a web proxy issue. We have a web proxy so I (and almost everyone in the company) has http_proxy and https_proxy set. I was able to get it to run by unsetting them as suggested by this grpc stackoverflow post. https://stackoverflow.com/a/68115391 which recommends passing the grpc.enable_http_proxy option.
Copy code
channel = grpc.insecure_channel('localhost:5005', options=(('grpc.enable_http_proxy', 0),))
However, it looks like the options to grpc.insecure_channel are hardcoded in DagsterGrpcClient. Not sure what the proxy impact would be on performance.
In case anyone else has this problem.... this was my issue. It resolved by setting no_proxy=127.0.0.1,localhost. Thumbs-upped the issue on github. https://github.com/dagster-io/dagster/issues/4297
4 Views