https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Antoine Valette

04/08/2022, 4:20 PM
Hello all, Thanks a lot for the great tool, super excited to see it grow that fast! I'm currently using Airflow with the Kubernetes Executor for some use cases. I strongly consider migrating because of some Airflow dependencies and design issues that bug me. Still there are a few things I like about Airflow that I'd like to keep if possible in Dagster w/ Kubernetes. After reading the documentation they are not clear to me. • Is it possible to specify a different container image for each
op
in a
job
or does it always use the same image as the
User Code Deployment
? ◦ We can override via the
K8sRunLauncher
or
k8s_job_executor
• Is it possible to run Dagster "out of cluster", have all the components running outside of Kubernetes (local docker compose for example) and only the
Run Workers
running in Kubernetes? ◦ It's possible to run the
K8sRunLauncher
out of cluster (
kubeconfig_file
arg) but not the
k8s_job_executor
from an other launcher. Would have loved to do so. • Are Kubernetes Job created at the Dagster
job
level (one Job per
job
), Dagster
op
level (one Job per
op
), or can we just configure one or the other? ◦
K8sRunLauncher
creates
job
level Jobs,
k8s_job_executor
creates
op
level Jobs. • Not sure I grasp the full difference between
K8sRunLauncher
and
k8s_job_executor
, I think it's linked to my point above? ◦
K8sRunLauncher
happens at
job
level,
k8s_job_executor
happens at
op
level. • What are the requirements for a custom
job
level or
op
level Docker image.
j

johann

04/08/2022, 9:08 PM
• Is it possible to run Dagster “out of cluster”, have all the components running outside of Kubernetes (local docker compose for example) and only the
Run Workers
running in Kubernetes?
◦ It’s possible to run the
K8sRunLauncher
out of cluster (
kubeconfig_file
arg) but not the
k8s_job_executor
from an other launcher. Would have loved to do so.
This can achieve what you described- if you configure the K8sRunLauncher and k8s_job_executor, they’ll handle k8s stuff while dagit/daemon can be wherever. The run workers do have deps on certain configmaps/secrets/etc. that our Helm chart provides, you’d have to copy them over (or modify the helm chart with a post render filter or something to remove dagit and the daemon)
• Not sure I grasp the full difference between
K8sRunLauncher
and
k8s_job_executor
, I think it’s linked to my point above?
K8sRunLauncher
happens at
job
level,
k8s_job_executor
happens at
op
level.
Exactly, so if you configure the
K8sRunLauncher
outside the cluster it will start the run worker inside a k8s job, then if you have the executor configured it will also run the ops in independent k8s jobs
• What are the requirements for a custom
job
level or
op
level Docker image. Here’s an example if using `dagster-k8s`: https://github.com/dagster-io/dagster/blob/master/python_modules/automation/automation/docker/images/dagster-k8s/Dockerfile
(Note if the image is only for jobs/ops, it doesn’t need Dagit)
a

Antoine Valette

04/09/2022, 10:32 AM
Hello @johann ! Thanks for the quick and precise answer 🙏 I've been playing with Dagster since yesterday and I can already say I'll be switching for our prod workloads this month.
9 Views