Hi I m having some unexpected behavior when trying to reload dagster #ask-community

Join Slack

Hi, I'm having some unexpected behavior when tryin...

# ask-community

Auster Cid

07/31/2023, 8:45 PM

Hi, I'm having some unexpected behavior when trying to reload a code location.

Auster Cid

07/31/2023, 8:47 PM

The webserver acts like the reload was successful, but nothing is actually reloaded. In dagster-grpc logs I see:

Copy code

ERROR:grpc._server:Exception calling application: Method not implemented!
2023-07-31T20:45:01.346960146Z Traceback (most recent call last):
2023-07-31T20:45:01.346971456Z   File "/usr/local/lib/python3.11/site-packages/grpc/_server.py", line 494, in _call_behavior
2023-07-31T20:45:01.346975126Z     response_or_iterator = behavior(argument, context)
2023-07-31T20:45:01.346978216Z                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-07-31T20:45:01.346981106Z   File "/usr/local/lib/python3.11/site-packages/dagster/_grpc/__generated__/api_pb2_grpc.py", line 290, in ReloadCode
2023-07-31T20:45:01.346984076Z     raise NotImplementedError("Method not implemented!")
2023-07-31T20:45:01.346987536Z NotImplementedError: Method not implemented!

Auster Cid

07/31/2023, 8:49 PM

Running `1.4.2`/`0.20.2` , on Docker with DockerRunLauncher

Auster Cid

07/31/2023, 10:22 PM

Just tried reproducing it with the

deploy_docker

example, same behavior there, but I don't see the error in the user code container's log. I'm defining my assets from an external config during code location startup and was counting on using this feature to externally trigger a reload. I'm wondering if assets being immutable within a docker image is dagster's expected behavior when deploying with docker?

Auster Cid

07/31/2023, 11:52 PM

https://github.com/dagster-io/dagster/discussions/14709 guess I should start by searching github discussions from now on 😛 thanks anyway o/

johann

08/01/2023, 1:51 AM

Glad you found that- to be clear the error comes up if you use the default

dagster api grpc

? We should handle that better

👍 1

Quentin Gaborit

08/01/2023, 12:23 PM

@johann jumping at the opportunity to ask you about system dependencies vs. code dependencies. In the above github discussion and the deployment guides it's clearly specified that user code can be separated from system code. However, I found out that my dependencies (e.g.

dagster-dbt

dagster-meltano

) should be installed on the daemon (i.e. system image) as much as it needs to be installed in my runner (code server image). Is my understanding incorrect?

johann

08/01/2023, 3:31 PM

If you’re deploying using

dagster api grpc

in a different container, then dagit and the daemon shouldn’t need access to user code or user dependencies. What’s the error you’re seeing? Could you share your workspace.yaml?

Quentin Gaborit

08/03/2023, 7:56 PM

@johann after a few correction it’s behaving as it should. My set up was incorrect because I didn’t expose the grpcserver correctly. Fixing that helped me deploying the helm chart as well. I’m now surprised that the code server is the one injecting environment variables in the k8 job pods? From my understanding it is the daemon that launches the runs isn’t it?

3 Views

Open in Slack

Previous Next