https://dagster.io/ logo
#announcements
Title
# announcements
n

Noah K

12/08/2020, 7:40 PM
On a scale of 1 to "are you kidding me?", how much pain am I in for trying to make a threaded executor? I've found one whammy already with some sys.path mutation but I think I can work around that.
m

max

12/08/2020, 8:57 PM
the default sqlite storages won't work with a multithreaded executor, you'll need to use the postgres storages
n

Noah K

12/08/2020, 8:58 PM
Already doing that 🙂
m

max

12/08/2020, 8:58 PM
be aware also that individual solid bodies will still execute in separate processes -- walking back the process isolation for user code will be hairy if you want to do that
n

Noah K

12/08/2020, 8:58 PM
Yes, I plan to do that eventually, that would be the threaded executor to come later
This is at the launcher level though for now
m

max

12/08/2020, 8:58 PM
yep
d

daniel

12/09/2020, 3:50 PM
@Noah K I think this actually might be somewhat painful to do at the launcher level - understanding more about the underlying goal here would be useful. It doesn't sound impossible, there just isn't a great pluggable part of the system to stick it in right now. (There are also some process isolation benefits you'd lose - since if I'm understanding the proposal correctly a bad run would now be able to mess up other runs or even crash the dagit process) A threaded executor (for a single run) would be great though and should be very pluggable, that's come up a bunch recently in other contexts
The issue is that I'm running the launcher worker in celery using celery's threaded mode
So I need to monkeypatch
alter_sys_path
to be a no-op (and ensure my modules are importable as-is)
d

daniel

12/09/2020, 8:51 PM
is it working other than that?
n

Noah K

12/09/2020, 8:53 PM
Yep 🙂
Just occasional global-state collisions when we run a lot of jobs at once
(two things trying to mess with sys.path in overlapped ways)
d

daniel

12/09/2020, 9:04 PM
ok, I think this would be our first run launcher that isn't doing each run in its own process - so I can't guarantee that you won't continue to run into some flakiness here, but as long as you're going into that clear-eyed that's fine with me :)
n

Noah K

12/09/2020, 9:05 PM
I wasn't able to find anything else messing with global state, but might be well hidden 😄
d

daniel

12/09/2020, 9:43 PM
I think we try to avoid it but it's hard to completely avoid sometimes - there are also some places where we do things with signal handlers
n

Noah K

12/09/2020, 11:00 PM
Ooh good call, will check for that in the launcher path