Hi everyone, I’m running a K8s deployment of dagster and I see that

multiprocess_executor

execute dagster job by k8s job so I wonder how is the

k8s_job_executor

different than the

multiprocess_executor

with

k8s_run_launcher

? Also how is

celetry_executor

different than

celery_k8s_job_executor

? And which one should I use on K8s? Is there any benchmark about these kinds of executor on K8s environment that I don’t know of?

multiprocess_executor

will create a subprocess per Op.

k8s_job_executor

will create a K8s Job per Op.

is there a performance advantage over either of these? we're currently investigating slow spin up times for some of our jobs.. sometimes taking over 90 secs from execution to start of the K8s Job.. would the multiprocess_executor be faster?

multiprocess would mean that all ops execute within one K8s Job, so you’d only pay the pod spinup cost once, compared to once per Op when using the

k8s_job_executor

so in a K8s ecosystem, we have one other pod that constantly runs, rather than having a new pod for each job??