On a scale of 1 to "are you kidding me?", how much pain am I in for trying to make a threaded executor? I've found one whammy already with some sys.path mutation but I think I can work around that.

the default sqlite storages won't work with a multithreaded executor, you'll need to use the postgres storages

Already doing that 🙂

be aware also that individual solid bodies will still execute in separate processes -- walking back the process isolation for user code will be hairy if you want to do that

Yes, I plan to do that eventually, that would be the threaded executor to come later

yep

@Noah K I think this actually might be somewhat painful to do at the launcher level - understanding more about the underlying goal here would be useful. It doesn't sound impossible, there just isn't a great pluggable part of the system to stick it in right now. (There are also some process isolation benefits you'd lose - since if I'm understanding the proposal correctly a bad run would now be able to mess up other runs or even crash the dagit process) A threaded executor (for a single run) would be great though and should be very pluggable, that's come up a bunch recently in other contexts

I already have my own launcher https://github.com/geomagical/dagster-geomagical/blob/main/dagster_geomagical/launcher.py

is it working other than that?

Yep 🙂

ok, I think this would be our first run launcher that isn't doing each run in its own process - so I can't guarantee that you won't continue to run into some flakiness here, but as long as you're going into that clear-eyed that's fine with me :)

I wasn't able to find anything else messing with global state, but might be well hidden 😄

I think we try to avoid it but it's hard to completely avoid sometimes - there are also some places where we do things with signal handlers

Ooh good call, will check for that in the launcher path