Hi all- Dagster now offers the

k8s_job_executor

, which executes solids in separate kubernetes jobs. This addition means that you can now choose at runtime between single pod and multi-pod isolation for solids in your run. Previously this was only configurable for the entire deployment- you could either use the K8sRunLauncher with the default executors (inprocess and multiprocess) for low isolation, or you could use the CeleryK8sRunLauncher with the celery_k8s_job_executor for pod-level isolation. Now, your instance can be configured with the K8sRunLauncher and you can choose between the default executors or the k8s_job_executor. Looking forwards to getting your feedback on this! Api docs: https://docs.dagster.io/master/_apidocs/libraries/dagster-k8s#dagster_k8s.k8s_job_executor Choosing an executor for your k8s deployment: https://docs.dagster.io/master/deployment/guides/kubernetes/deploying-with-helm#executor

nice! How does

k8s_job_executor

compare to the Celery executor?

Previously, the

celery_k8s_job_executor

was the only way that dagster supported job isolation on kubernetes. With the new

k8s_job_executor

, we want to offer a simpler way to achieve that for most users who don’t need all of Celery’s features (and overhead). Both offer the same isolation of running each step in a separate k8s job.

celery_k8s_job_executor

requires you to deploy a standing set of Celery workers, which manage the steps.

k8s_job_executor

doesn’t require extra workers.

celery_k8s_job_executor

offers some features for advanced Celery users that we don’t support yet in the `k8s_job_executor`: priorities, separate step queues with concurrency limits, etc.

Thank you!