https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Don

06/16/2023, 6:55 PM
I started getting this message today:
Copy code
Agent ******** detected that run worker failed: The code location server that was hosting this run is not responding. It may have crashed or been OOM killed.
Is this an issue with Dagster Cloud, or my code? We haven't changed anything in our deployment. Thanks.
d

daniel

06/16/2023, 8:11 PM
Hi Don - is this using non-isolated runs? https://docs.dagster.io/dagster-cloud/deployment/serverless#non-isolated-runs If so my recommendation would be to try it using the default isolation level where each run has its own ECS task - those runs have a lot more memory to work with
d

Don

06/16/2023, 8:22 PM
Thanks for the help. I tried running both isolated and non-isolated with no luck. Are there any issues with cloud right now? We have never seen this before.
d

daniel

06/16/2023, 8:23 PM
Can you link to a run that is having the problem with an non-isolated isolated run? We can take a look in our logs. There aren't any current issues with cloud that i'm aware of
er sorry, with an isolated run
d

Don

06/16/2023, 8:24 PM
ok, I'll try it again. I noticed that a new agent was deployed 5hrs ago with version: 1.3.10 The previous Agent (that was working) was version: 1!0+dev
d

daniel

06/16/2023, 8:24 PM
I wouldn't expect that to matter - if you can pass along a link I'd be happy to take a look
d

Don

06/16/2023, 8:25 PM
Isolated run does this:
Copy code
dagster._core.errors.DagsterUserCodeUnreachableError: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE

Stack Trace:
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 797, in _process_api_request
    api_result = self._handle_api_request(
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 580, in _handle_api_request
    serialized_snapshot_or_error = client.execution_plan_snapshot(
  File "/dagster/dagster/_grpc/client.py", line 221, in execution_plan_snapshot
    res = self._query(
  File "/dagster/dagster/_grpc/client.py", line 157, in _query
    self._raise_grpc_exception(
  File "/dagster/dagster/_grpc/client.py", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
It won't even start
d

daniel

06/16/2023, 8:25 PM
Can you send the name of your Dagster Cloud organization?
d

Don

06/16/2023, 8:26 PM
I will message you
3 Views