Hi I m using the `k8s job executor` and specifying `tolerati

Question

Hi I m using the `k8s job executor` and specifying `tolerations` `nodeSelector` tags so that pods will run on a specific node group This works as expected for the `run` pod where I can see that my `tags` were properly applied in the pod s yaml file and the pod runs on a node in the correct node group however the `step` pods do not execute on the correct node group and when I examine the `step` pod yaml file the `tolerations` `nodeSelector` tags are not present Is this the expected behavior Is there another way to add tags to individual step pods

Nolan Nichols · Answer

Here is how I am configuring my asset job ```metsim assets job = dagster define asset job name= metsim asset job selection=dagster AssetSelection groups metsim executor def=k8s job executor tags= dagster k8s config container config resources requests cpu 8000m memory 10Gi limits cpu 8000m memory 10Gi pod template spec metadata annotations <http cluster autoscaler kubernetes io safe to evict|cluster autoscaler kubernetes io safe to evict> true pod spec config tolerations key mazeApp value dagster effect NoSchedule nodeSelector AppUsed dagster ```

Nolan Nichols · Answer

related thread <https dagster slack com archives C014N0PK37E p1666707971905489>

daniel · Answer

Hey Nolan for it to apply to the step pods I believe you d want those same tags to be on each asset since each asset maps to a step pod

Nolan Nichols · Answer

Thanks Daniel that makes sense I m using `dagster AssetsDefinition from graph`so then for each `op` in my graph I would add the same tags that I am currently adding just at the `dagster define asset job` level

daniel · Answer

That s my understanding yeah bit verbose but the tags can of course go in a separate variable

Nolan Nichols · Answer

Yup I mean it makes sense given that a step maps to an op I ll give it a shot and let you know how it goes

Nolan Nichols · Answer

Worked like a charm thank you

Nolan Nichols · Answer

Note that I missed the docs for this I was expecting some mention of this in the apidocs for `k8s job executor` so I added a note to the doc string in this merged