https://dagster.io/ logo
h

Hebo Yang

04/12/2021, 10:12 PM
Hi team, I read that default-run-launcher spawns a new process on pipeline’s repository nodes. Does it mean the “Run worker” in this case will also be a process on the node instead of a K8s job please?
d

daniel

04/12/2021, 10:16 PM
Hi Hebo, that's right - you can use the K8sRunLauncher or the CeleryK8sLauncher if you'd like the run worker to be a K8s job
The Kubernetes docs have some examples where the run worker is a K8s job here: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm
h

Hebo Yang

04/13/2021, 3:33 AM
Hi Daniel! Thanks so much for the reply! I am actually trying to avoid spinning up new pods as our infra currently is solely Terraform based and doesn’t support dynamically creation of jobs/pods.
I am planning to use default-run-launcher first and then maybe explore a way to use a custom launcher to use a fixed pool of pods.