Is there documentation on how the

in_process_executor

should behave? having set

Definitions(executor=in_process_executor)

I was expecting all my assets and partitions to materialize in the same process, but yet, each partition and each asset still runs in its own process. I double checked the logs and each run had its own different PID

The

in_process_executor

is for making sure all ops or assets within a single job run execute within a single process. Different runs will use different processes - Dagster doesn't keep a thread alive within which each run is launched. Partitions seem to generally be calculated in different runs, unless you're doing a backfill where you send multiple partitions to a single run. There may be other ways to yield multiple partitions from a single run but I'm not super knowledgable on partitions.

I see thanks, Is there a "threaded executor" which will execute all ops and materializations within the "server" always up process? to avoid latency.

Not to my knowledge - I think that due to the GIL something like that would block the server process from serving other job run requests. If latency is your main concern you could use the DefaultRunLauncher if you're not already, that run launcher will at least run your jobs on the same server that handles the requests, instead of launching an external process like the K8s run launcher or the ECS run launcher

Thanks for sharing your experiences, @Zach. Do you (or dagster) have recommendations, which options exist to reduce launch latency? What do you think is the minimum periodicity one could aim for with dagster?

I'm probably not the best person to ask about this and I don't want to overspeak my experience as my work with Dagster has been mostly with very large very long jobs where initialization latency is not really a concern. However, I hang out on this slack channel a lot to learn from others and have anecdotally seen a number of folks who are trying to use Dagster in a low latency contexts struggling. However I know that performance has been something the Dagster team has been focused on recently, particularly around sensor evaluations. It seems that initialization penalties are mostly dependent on a combination of the type of RunLauncher you're using, how big the instance is that your user code locations run on, and how large your code is / how long it takes for your code to load into a process (lots of top-level imports / things executed at the global level would have an effect here). I think I've seen @Stephen Bailey has had some success with relatively quick spinup (~10 seconds) for jobs using the kubernetes run launcher. The EcsRunLauncher is inherently slow because ECS/ECR doesn't provide any image caching (roadmap item on AWS for years now, I'll shit a gold brick before they implement it), so your image needs to be pulled down for every job run. When I was self-hosting it seemed like the DockerRunLauncher was pretty fast, as is the DefaultRunLauncher. I've never experiemented with the CeleryK8sRunLauncher, but it seems plausible that it could be quite fast if there are already celery workers available I don't really know what a possible minimum periodicity would be, but I suspect getting into sub-second territory would be very difficult if not impossible without deep customization. A few seconds seems achievable, as long as you're not in a high-volume scenario - I've also seen anecdotes about the queue coordinator struggling to keep up with high-volume jobs