https://dagster.io/ logo
Join Slack
Powered by
Hello everyone, am I correct in understanding that...
# deployment-kubernetes
n
Hello everyone, am I correct in understanding that dagster-job is for user tasks, while dagster-step is for user operations and assets? However, I don’t have any tasks (jobs), only assets. But for some reason, dagster-job is appearing. Is there a way to disable dagster-job?
I forgot to mention that I’m running with the CeleryK8sRunLauncher.
j
dagster-job-…
k8s jobs are run workers. These schedule the compute for underlying Ops or Assets in a run. 
dagster-step-…
is a step worker. A step = the execution of an Op or Asset. These get kicked off by their respective run workers. https://docs.dagster.io/deployment/overview#job-execution-flow
In the event that you are often materializing only one asset, this does mean you have the extra overhead of the 
dagster-job…
k8s job that doesn’t actually do much. My recommendation in that case is to use the K8sRunLauncher and the multiprocess_executor (default). This way you’ll only have 
dagster-job…
k8s jobs, and the asset will execute inside of it.
n
Hello Johann, thank you for your response. We are currently using K8sRunLauncher in one cluster but would like to switch to CeleryK8sRunLauncher. The main issue lies with the nodeSelector for the dagster-run-* jobs as they are not being scheduled on the desired nodes.
j
It’s possible to set nodeSelector on the run launcher, which will apply to dagster-run- jobs https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#instance-level-kubernetes-configuration
Open in Slack
PreviousNext
Powered by