Hello I ve been using Dagster to ETL small partitions of a d

Question

Hello I ve been using Dagster to ETL small partitions of a dataset Each ETL step is an op I ve noticed it takes dagster about 15 seconds to launch a subprocess for each op Is this expected I m using the default run launcher and executor

johann · Answer

Hi Bennett with 0 14 0 releasing on Thursday we re planning to change the behavior of the default executor to use `forkserver` to start subprocesses This will significantly reduce the overhead between Ops

johann · Answer

Until then you might consider using the `in process` executor which will run your Ops serially but won t have the overhead of creating new processes

Bennett Norman · Answer

Thank you Looking forward to experimenting with the change