The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hi dagsterfans, I have a sensor that kicks off many small jobs that each take less then a minute to complete. however it takes up to a minute for the dagster to realize that the job is finished. Is there a way to speed that up? (I'm using run monitoring, using the k8s job executor (one container per job) )

I noticed the termination of subprocesses (I use the multiproces executor) takes about a minute too.

finished steps in process in 1m1 seconds. (can't we make that faster?)