I started getting this message today:

Agent ******** detected that run worker failed: The code location server that was hosting this run is not responding. It may have crashed or been OOM killed.

Is this an issue with Dagster Cloud, or my code? We haven't changed anything in our deployment. Thanks.

Hi Don - is this using non-isolated runs? https://docs.dagster.io/dagster-cloud/deployment/serverless#non-isolated-runs If so my recommendation would be to try it using the default isolation level where each run has its own ECS task - those runs have a lot more memory to work with

Thanks for the help. I tried running both isolated and non-isolated with no luck. Are there any issues with cloud right now? We have never seen this before.

Can you link to a run that is having the problem with an non-isolated isolated run? We can take a look in our logs. There aren't any current issues with cloud that i'm aware of

ok, I'll try it again. I noticed that a new agent was deployed 5hrs ago with version: 1.3.10 The previous Agent (that was working) was version: 1!0+dev

I wouldn't expect that to matter - if you can pass along a link I'd be happy to take a look

Isolated run does this:

dagster._core.errors.DagsterUserCodeUnreachableError: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE

Stack Trace:
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 797, in _process_api_request
    api_result = self._handle_api_request(
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 580, in _handle_api_request
    serialized_snapshot_or_error = client.execution_plan_snapshot(
  File "/dagster/dagster/_grpc/client.py", line 221, in execution_plan_snapshot
    res = self._query(
  File "/dagster/dagster/_grpc/client.py", line 157, in _query
    self._raise_grpc_exception(
  File "/dagster/dagster/_grpc/client.py", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(

Can you send the name of your Dagster Cloud organization?

I will message you