https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
n

Nikita Kolodeznov

06/22/2023, 8:01 AM
Hello everyone, am I correct in understanding that dagster-job is for user tasks, while dagster-step is for user operations and assets? However, I don’t have any tasks (jobs), only assets. But for some reason, dagster-job is appearing. Is there a way to disable dagster-job?
I forgot to mention that I’m running with the CeleryK8sRunLauncher.
j

johann

06/22/2023, 2:14 PM
dagster-job-…
k8s jobs are run workers. These schedule the compute for underlying Ops or Assets in a run.
dagster-step-…
is a step worker. A step = the execution of an Op or Asset. These get kicked off by their respective run workers. https://docs.dagster.io/deployment/overview#job-execution-flow
In the event that you are often materializing only one asset, this does mean you have the extra overhead of the
dagster-job…
k8s job that doesn’t actually do much. My recommendation in that case is to use the K8sRunLauncher and the multiprocess_executor (default). This way you’ll only have
dagster-job…
k8s jobs, and the asset will execute inside of it.
n

Nikita Kolodeznov

06/23/2023, 12:05 PM
Hello Johann, thank you for your response. We are currently using K8sRunLauncher in one cluster but would like to switch to CeleryK8sRunLauncher. The main issue lies with the nodeSelector for the dagster-run-* jobs as they are not being scheduled on the desired nodes.
j

johann

06/23/2023, 3:50 PM
It’s possible to set nodeSelector on the run launcher, which will apply to dagster-run- jobs https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#instance-level-kubernetes-configuration