https://dagster.io/ logo
Join Slack
Powered by
what is the best way for me to manage kubernetes r...
# ask-community
s
what is the best way for me to manage kubernetes resource allocation at the job level? For example, if I am going to run a 
dbt_cloud
job that is just going to be polling an api for 2 hours, how could i specify a lightweight pod? Should I define k8s_job_executor and apply some sensible defaults to it for the job, a la https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.k8s_job_executor?
and will this get ignored for local development? I think this is the case, but i want to make sure before going deeper
j
Hi Stephen, you can use tags on the specific op to override resource requirements https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment
will this get ignored for local development?
You should have a version of the jobs without the k8s executor for local dev. So it won’t have the k8s run config, and the tags will be ignored
s
in that example in the docs, it doesn't look like the 
k8s_job_executor
is specified at all. in that case, will the tags just be ignored at run time in local environments?
because then i could just have a couple of profiles like
Copy code
from local.profiles import lightweight_k8s_profile

@job(tags=lightweight_k8s_profile)
def foo():
   bar_op()
j
That’s a good call out, the docs are a bit misleading there. If the 
k8s_job_executor
is configured, the tags for each op will be used. If the executor isn’t configured, all of the ops will be executing within a single K8s Job, so only the tag on the overall job is used
👌 1
2 Views
Open in Slack
PreviousNext
Powered by