https://dagster.io/ logo
Powered by
Title
m

madhurt

06/30/2021, 9:39 PM
Hi, quick question. I currently use 
k8sLauncher
. It creates separate jobs for each pipeline run. Now, just like one can assign pods to a specific worker node using the 
nodeSelector OR affinity OR tolerations
properties. Can I assign each job created by the 
k8sLauncher
to a specific node? I don’t see these properties for 
k8sLauncher
in 
.Values
thingie by 
helm
. But I do see these properties for 
celeryK8sRunLauncher
. Does that mean I need to switch to ``celeryK8sRunLauncher``?
r

rex

06/30/2021, 11:02 PM
This is not currently exposed to the k8sLauncher as a global property. But you can get the same effect by adding tags to your pipelines to ensure that they are assigned to specific worker nodes
https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#solid-or-pipeline-kubernetes-configuration
m

madhurt

07/02/2021, 6:56 PM
I see, so applying tags to the pipelines i.e. 
nodeSelector
should ensure that jobs launched by the 
k8sLauncher
go to that specific node? (Notice that I want to send jobs to a node, not the user deployment image pod). Also, can we apply such tags to a 
repository
and 
sensors
as well? or only 
pipelines
and 
solids
?
unfortunately, I get 
TypeError: __init__() got an unexpected keyword argument 'nodeSelector'
when I use 
tags={
              'dagster-k8s/config': {
                  'pod_spec_config': {
                      'nodeSelector': {
                          "jobs": "true"
                      }
                  }
              }
          }
I think it’s called 
node_selector
, will try again and let you know if this works
I think it worked. Thanks a lot. There are just two questions that I have now: 1) How can I use the 
nodeSelector
property for the 
dagster-postgres
pod? I couldn’t find it in the global 
.Values
thingie for YAML and even if I did in the 
.Values
specific to 
postgres
, I found it in 2 places under 
Postgres master
and 
Postgres slave
. Do I set it 2 times for master and slave? 2) Can I apply these tags to 
sensors, repository
and 
schedules
as well?
cc: @rex @alex @johann
r

rex

07/06/2021, 10:50 PM
1. Unless you have have replication enabled for your postgres chart, you should be fine with just setting the node selector for postgres master. 2. sensors/schedules are based on an underlying pipeline, so you can add the tags to that pipeline, if that’s what you’re asking. Repositories are not tagged. A set of repositories is exposed in a user code deployment - the user code deployment has its own node selector that can be configured in the helm chart.
#dagster-supportJoin Slack
Powered by