Xu Zhang
11/16/2020, 4:19 AMdagster.serdes.ipc.DagsterIPCProtocolError: Timeout: read stream has not received any data in 15 seconds
it could be possibly caused by how long the deployment process takes which exceeds the timeout defined.
The error was raised from wait_for_grpc_server
, however, I did explicitly add below config to dagster.yaml, why it still tried to start a gRPC server?
opt_in:
local_servers: false
stacktrace:
/export/apps/python/3.7/bin/python3.7: Error while finding module specification for 'dagster.grpc' (ModuleNotFoundError: No module named 'dagster')
Loading repository...
KafkaClient cannot be created for Log Handler in DEV fabric. No events will be sent!
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/nbformat/notebooknode.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
from collections import Mapping
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
from collections import MutableMapping, Sequence # noqa
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
from collections import MutableMapping, Sequence # noqa
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/graphene/relay/connection.py:2: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
from collections import Iterable, OrderedDict
/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/cli/workspace/workspace.py:50: UserWarning: Error loading repository location dagsterweb:(DagsterIPCProtocolError) - dagster.serdes.ipc.DagsterIPCProtocolError: Timeout: read stream has not received any data in 15 seconds
Stack Trace:
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/cli/workspace/workspace.py", line 43, in _load_handle
handle = RepositoryLocationHandle.create_from_repository_location_origin(origin)
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/core/host_representation/handle.py", line 47, in create_from_repository_location_origin
return ManagedGrpcPythonEnvRepositoryLocationHandle(repo_location_origin)
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/core/host_representation/handle.py", line 172, in __init__
lazy_load_user_code=True,
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/grpc/server.py", line 1095, in __init__
lazy_load_user_code=lazy_load_user_code,
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/grpc/server.py", line 1008, in open_server_process
wait_for_grpc_server(output_file)
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/grpc/server.py", line 955, in wait_for_grpc_server
event = read_unary_response(ipc_output_file, timeout=timeout)
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/serdes/ipc.py", line 39, in read_unary_response
messages = list(ipc_read_event_stream(output_file, timeout=timeout))
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/serdes/ipc.py", line 142, in ipc_read_event_stream
timeout=timeout
location_name=location_name, error_string=error_info.to_string()
Exception ignored in: <function ManagedGrpcPythonEnvRepositoryLocationHandle.__del__ at 0x108704950>
Traceback (most recent call last):
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/core/host_representation/handle.py", line 245, in __del__
self.is_cleaned_up,
File "/export/content/lid/apps/dagster-web/i001/libexec/dagster-web_e1306745056a4ad8d3f2195a879de5b31aa2840bd78ffce543abb9b3e57d908a/site-packages/dagster/core/host_representation/handle.py", line 241, in is_cleaned_up
return not self.client
AttributeError: 'ManagedGrpcPythonEnvRepositoryLocationHandle' object has no attribute 'client'