Hi I am running into an unexpected error using dagster serve dagster #ask-community

Hi - I am running into an unexpected error using d...

David Merritt

01/12/2023, 7:52 PM

Hi - I am running into an unexpected error using dagster-serverless:

Copy code

Multiprocess executor: child process for step historical_fire_perimeters_geojson unexpectedly exited with code -9
dagster._core.executor.child_process_executor.ChildProcessCrashException

David Merritt

01/12/2023, 7:53 PM

It seems to occur after a successful yield of an

op

result in a job, halting the flow of the job.

David Merritt

01/12/2023, 7:53 PM

cc @Gabe Schine - trying out your branch

jordan

01/12/2023, 7:58 PM

Generally that’s an out of memory error. You might be launching the run in “non-isolated” mode meaning it’s sharing resources with some of your other processes. Switching to “isolated” mode might help get around this.

jordan

01/12/2023, 7:58 PM

https://docs.dagster.io/dagster-cloud/deployment/serverless#run-isolation

David Merritt

01/12/2023, 8:18 PM

hmm let me try that, but it seems like the isolated run takes 3+ minutes extra to start?

David Merritt

01/12/2023, 8:22 PM

Any idea why that would happen after a successful yield on an op? I wouldn’t expect > 1gB of memory usage in that op. It’s downloading/parsing some json but its not THAT much data.

Gabe Schine

01/12/2023, 8:23 PM

I think it's on the order of 50mB, IIRC

David Merritt

01/12/2023, 8:24 PM

Copy code

They have fewer compute resources: 0.25 vCPU cores and 1GB of RAM. These resources are shared with other processes for a code location like sensors. As a result, it's recommended to use isolated runs for compute intensive jobs and asset materializations.

David Merritt

01/12/2023, 8:24 PM

50mB of json @Gabe Schine? even if that doubles when parsed into the FS model, we’re well under gB but I’m not sure the basic overhead of the dagit runner/baseline level

David Merritt

01/12/2023, 8:25 PM

isolated run did fix the issue thanks @jordan

Gabe Schine

01/12/2023, 8:25 PM

Yes, 50mB of uncompressed JSON string

David Merritt

01/12/2023, 8:26 PM

looks like ~1minute of spin up time, I think that’s acceptable. Looks like this won’t be an issue for scheduled jobs @Gabe Schine.

Copy code

When launching runs manually, select Isolate run environment in the Launchpad to launch an isolated runs. Scheduled, sensor, and backfill runs are always isolated.

Gabe Schine

01/12/2023, 8:27 PM

yep my read of the docs as well. does the UI let you do a manual asset materialization but isolate its job?

David Merritt

01/12/2023, 8:29 PM

yes - checkbox in the “materialization config” modal.

✔️ 1

62 Views

Open in Slack

Previous Next