https://dagster.io/ logo
Join Slack
Powered by
Hi, recently I've been experimenting with the dags...
# announcements
e
Hi, recently I've been experimenting with the dagster-daemon and an error related to the scheduler comes up now and then that basically resolves itself after some time and a refresh. Is this a bug or am I missing something?
The error is as follows:
Copy code
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNKNOWN
	details = "Exception iterating responses: cannot import name 'EnvironmentClassCreationData' from partially initialized module 'dagster.core.definitions.environment_configs' (most likely due to a circular import) (/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/core/definitions/environment_configs.py)"
	debug_error_string = "{"created":"@1617748195.680820000","description":"Error received from peer unix:/var/folders/yn/m0pvh2vd52g46p8sk37zkkhc0000gn/T/tmpuvjplgjx","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Exception iterating responses: cannot import name 'EnvironmentClassCreationData' from partially initialized module 'dagster.core.definitions.environment_configs' (most likely due to a circular import) (/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/core/definitions/environment_configs.py)","grpc_status":2}"
>
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/scheduler/scheduler.py", line 93, in launch_scheduled_runs
    repo_location = location_manager.get_location(origin)
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/core/host_representation/location_manager.py", line 49, in get_location
    else GrpcServerRepositoryLocation(
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/core/host_representation/repository_location.py", line 480, in __init__
    self._external_repositories_data = sync_get_streaming_external_repositories_data_grpc(
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/api/snapshot_repository.py", line 15, in sync_get_streaming_external_repositories_data_grpc
    external_repository_chunks = list(
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/grpc/client.py", line 245, in streaming_external_repository
    for res in self._streaming_query(
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/dagster/grpc/client.py", line 95, in _streaming_query
    yield from response_stream
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/grpc/_channel.py", line 416, in __next__
    return self._next()
  File "/Users/synergia/.virtualenvs/dagster/lib/python3.9/site-packages/grpc/_channel.py", line 803, in _next
    raise self
d
very strange! I don't remember another report with this error. It's particularly strange that it's sporadic. Do your pipelines/schedules do anything that alter sys.modules or might affect the python module loading path?
if its possible to share your repository code we could take a look and see if anything jumps out
would you also mind sharing what version of dagster you're using?
e
Sorry for the late response. After watching it for a while, I discovered that the one returning the error is sometimes the sensor daemon and other times the scheduler daemon, and it almost always occurs at startup (after using 
dagster-daemon run
). From what I know, my pipeline is still very basic, so I don't think that me modifying the module loading path is not even a possibility. I'm using Dagster 0.11.3
For my code: Do you need my entire script or only the repo part?
d
If it's possible to provide the repository and pipelines/schedules that would be helpful so we can try to repro, but only whatever you're comfortable sharing. One other question - is there anything else suspicious in the logs earlier / any other error-like output that doesn't necessarily stop execution?
e
Here's my entire script. It's probably a bit janky and long, but I hope it helps!
Tests.py
And... nope. I don't see any other warnings or suspicious logs before the aforementioned error pops up.
d
Got it, thanks - I have the daemon running with the schedule running and haven't managed to run into the error yet. Are there any steps that you've found can reliability reproduce the problem?
e
The problem seems to pop up every time I launch the dagster-daemon.
d
Got it - so you see it 100% of the time, whenever you execute "dagster-daemon run"?
but then it sorts itself out after some time?
e
So far, yes. After that it continues to appear sporadically and quickly fixes itself. This is to say that my sensors and schedules work right despite the warning.
d
If you execute the Tests.py file in python directly do you also get the issue?
I guess that is unlikely to be the problem since the error refers to a dagster file
would you mind sharing your workspace.yaml as well?
ah wait, I can reproduce once I turn both the schedule and the sensor on!
e
Ohhh! Maybe a conflict between the two? I don't get the issue when I execute it directly on python... and sadly I don't have a workspace.yaml file in my directory.
d
What command did you run to start dagit?
e
dagit -f Tests.py
d
Got it - I think I see the problem. Thanks for the report, this is very helpful. I bet we can get this fixed for our weekly release tomorrow
e
Glad I can help!
d
Wow I think this might actually be a python bug in python 3.9! https://bugs.python.org/issue43517
looks like they have fix in cpython landed in master. Is using 3.8 an option until they ship the next version?
e
Wooow. Did not expect that
I suppose I can switch to python 3.8 when I enter production
Thanks a lot @daniel!
condagster 1
3 Views
Open in Slack
PreviousNext
Powered by