Hey all – I'm seeing a `dagster._core.executor.child_process_executor.ChildProcessCrashException`every time I run an asset materialization for the first time after a server restart. It's not an OOM issue – I'm processing on the order of ~200MB of data and the container has 32GB. Every time after retrying a second time it succeeds. What could be the cause of this / how can I dig in further?

Hi Kevin - do you have a full stack trace from the error?

hey @daniel this is all i get unfortunately in the web UI

Multiprocess executor: child process for step seatgeek_events_raw was terminated by signal 7 (SIGBUS).

dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:

File "/opt/conda/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 240, in execute

event_or_none = next(step_iter)

,  File "/opt/conda/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 357, in execute_step_out_of_process

for ret in execute_child_process_command(multiproc_ctx, command):

,  File "/opt/conda/lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 174, in execute_child_process_command

raise ChildProcessCrashException(exit_code=process.exitcode)

SIGBUS is unusual.. any chance you’re able to share code that reproduces the problem? The crash is very likely coming from the code in the body of the op - are you able to reproduce the crash when running the same code or dagster job locally?

@daniel I wasn't getting anything similar to this when I was running locally, this started when I followed the guide to get Docker working. Now it works sometimes and sometimes it doesn't. Using the default run launcher right now, i saw a bunch of errors when i used the Docker run launcher.

one thing you could do is try running it with the in_process executor (that runs everything in the same process) instead of the default multiprocess executor that runs each op in its own subprocess (e.g. by adding this to your run config in the launchpad

execution:
  config:
    in_process:

i'd still expect it to crash but maybe there might be more information about the crash since it'd be happening in the 'main' process? or if it doesn't crash that might be a clue

yeah I haven't been able to reproduce a crash reliably yet because I'm not getting any detailed errors that tell me what failed. I'm using assets for everything, do assets work with the launchpad? can't find it

Yeah, it's a bit hidden but if you shift click on the text of the Materialize button it will open the launchpad

oh nice ok let me try

I think null is fine

is there a way to make the log output more verbose maybe

like specify a global python log level for the libraries accessed within the op you mean? You can add additional python loggers and configure their logging level here: https://docs.dagster.io/concepts/logging/python-logging#capturing-python-logs-

i'm already doing some logging with

<http://context.log.info|context.log.info>

and similar in my ops – I meant is there a way for us to dig into what's actually causing the signal 7 SIGBUS thing. Like when I was running locally if anything failed, I'd always get a full stacktrace

Yeah with most failures I would expect the stack trace to appear in Dagit and the container output - crashes are trickier, I wouldn't expect Dagster to make that any harder to diagnose than it would be in any other Python program but it can still be very annoying to track down

@daniel I did

faulthandler.enable()

and unfortunately I get the same thing

dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/opt/conda/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 240, in execute
    event_or_none = next(step_iter)
,  File "/opt/conda/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 357, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
,  File "/opt/conda/lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 174, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)

Maybe try with the in process executor and faulthandler.enable?

that one gave me even less information, no stack trace

Nothing in the raw process logs for dagit in the terminal either?

nothing in the raw process logs for any of the containers other than what I pasted above @daniel – dagit seems happy

2023-02-25 14:53:24 2023-02-25 19:53:24 +0000 - dagit - INFO - Serving dagit on <http://0.0.0.0:3003> in process 1

like I mentioned, this op was working fine when I just ran

dagster dev

locally but in Docker it only works sometimes

the assets version of execute_in_process is `materialize`: https://docs.dagster.io/_apidocs/execution#materializing-assets

ok gotcha – is that behavior different from when when we executed with the launchpad config like this?

execution:
  config:
    in_process:

the main difference is it will be in memory in the process that calls the function - it won't go through the run launcher

that worked @daniel, I was able to do by calling the op directly and through materialize

@asset(group_name="unified", retry_policy=retry_policy)
def events(context: OpExecutionContext):
    # ...
    # op definition
    # ...

# works
events(None)

# also works
materialize([events])

I did see that, I think that would be really great

I think the issues I'm hitting in this thread might be resolved if the actual code runs in another container rather than in the same one as the GRPC server

Using the DockerRunLauncher seems like a good step, yeah - should if nothing else isolate things quite a bit more