I would love to use the multiprocess executor's start_method: forkserver (https://docs.dagster.io/concepts/ops-jobs-graphs/job-execution#default-job-executor) but have hit crashes with it in the past. Is there a tracking bug for improving stability there? Is https://github.com/dagster-io/dagster/issues/4041 the right issue to watch? Are the issues with certain libraries something Dagster could potentially solve, or is that up to the underlying Python multiprocess library?

cc @alex

theres generally not much we can do about libraries crashing when

fork

-ed, so theres no ongoing stability work. You can certainly report the issues you are seeing and we can see if we can help.

I got

start_method: forkserver

to work on MacOS X by following https://stackoverflow.com/questions/50168647/multiprocessing-causes-python-to-crash-and-gives-an-error-may-have-been-in-progr and setting:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
export no_proxy=*

Along with a

preload_modules

of my job's main entrypoint (so it effectively loads everything I'll need), this gave about a 2x speedup for local Dagster runs, which is great!

there is not - can you use

SimpleSpanProcessor

in stead? How many spans are you generating?

Thanks! Yes,

SimpleSpanProcessor

seems to work fine at least locally. A local/test pipeline run might generate 55 spans over 30s, and a full pipeline run generates about 30k over 5-6h.