https://dagster.io/ logo
Title
s

Son Giang

11/09/2022, 8:51 AM
Hi team, what is the difference between
celery_executor
and
celery_k8s_job_executor
?
d

daniel

11/09/2022, 8:20 PM
Hi Son - the celery_k8s_job_executor will launch each op as a kubernetes pod: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm-advanced - the celery_executor will run the op from the celery worker itself i believe
s

Son Giang

11/10/2022, 3:19 AM
what is the benefit of running each op as a kubernetes pod vs running it in the celery worker itself?
d

daniel

11/10/2022, 3:19 AM
Kubernetes gives you things like autoscaling
👍 1
And isolation between ops
👍 1
s

Son Giang

11/10/2022, 3:35 AM
Thanks @daniel, the
k8s_job_executor
also launch ops in k8s pod. As I understand right now, if you use
k8s_job_executor
and
k8s_run_launcher
, the daemon will handle the spawning of k8s pod, but in
celery_k8s_job_executor
and
celery_k8s_run_launcher
, the celery workers will handle the spawning of k8s pod. Is that a correct understanding? And what is the benefit of
celery_k8s
over
k8s
? Is the
celery_k8s
more stable?
d

daniel

11/10/2022, 3:36 AM
The doc I posted earlier explains the requirements that would lead you to pick celery-k8s (per-op limits and prioritization rules across multiple jobs)
If that’s not a requirement you have, I would not recommend using celery
s

Son Giang

11/10/2022, 3:41 AM
I see