19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
s
Slackbot
04/05/2023, 12:11 PMThis message was deleted.
d
daniel
04/05/2023, 12:14 PMHi Sundara - can you post the output of 'kubectl describe <pod name>' for a dagster-step pod with this config for one of your steps?
I'm not totally following your comment about the multiprocess executor since that wouldn't be putting each op in its own k8s pod. There would just be one pod for the whole run, and that would pull any config from the job, not from anything on your ops.
the main thing i would want to check from kubectl describe is whether the tolerations that you configured are acutally being applied - are you seeing what you expect when you inspect the pod?
So to confirm - the issue now is that setting tolerations via run_k8s_config is having no effect in the run pod? If you kubectl describe it, you don't see any tolerations being applied? Do other things that you set via run_k8s_config work?
What version of Dagster are you using?
Earlier you said "But if i change it to multiprocess executor, the pod's are getting deployed on the correct node" - I'm having trouble matching that with the most recent comment that the tolerations aren't being applied
Is it all run_k8s_config fields that aren’t being applied, or just tolerations?
Are you certain that the image that’s launching the run is also on dagster 1.1.14 - this is using the Dagster helm chart? Asking because run_k8s_config was added fairly recently and wouldn’t be applied on older versions
Can you give a set of steps to reproduce the problem using your setup? I’m pretty confident that run_k8s_config works in general when using the helm chart, so there would have to be something unique about how you’re launching runs using it
I see - my understanding is that execute_job doesn't go through the run launcher or create a new pod. It executes the job synchronously in the same process that calls the execute_job function.
so i'm not totally clear which pod you would be checking for tolerations for the run
the executor won't change whether or not execute_job uses the run launcher - it will change the behavior for how each op within the run is executed
i.e. it would allow you to configure the step pods but not the run pod (since there will never be a run pod)
I’m not totally sure what “a k8s-job.yaml” refers to exactly, can you elaborate?
I see - so you’ve kind of implemented your own run launcher here it sounds like
You can use the k8s job executor while running execute_job and it will spin up pods for each step, yes
I would have expected the tags in your original post to work - can you post the full op decorator with the tags?
similar question to before - does all pod_spec_config / dagster-k8s/config not work, or just the tolerations? Do you see that ttl_seconds_after_finished that you also set being applied to the k8s job that's created for each step?
Can you pass along a full set of code that reproduces the problem?