https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
f

fahad

02/15/2023, 8:34 PM
Hey everyone, is there a way to define the default k8s resources on all asset materialization jobs without having to define a job from each asset and manually applying the k8s tags there?
d

daniel

02/15/2023, 8:37 PM
Hey fahad - we have a way to set this for all jobs, but I don't think we have a way to specifically differentiate asset materialization jobs from other jobs, would that still help here?
f

fahad

02/15/2023, 8:37 PM
Being able to set a global default for all jobs would be good enough for my use-case!
(or rather per Code Location/Definition)
d

daniel

02/15/2023, 8:40 PM
For per-code-location I think the resources key here would affect every pod spun up for that code location https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#step-61-configure-the-deployment
f

fahad

02/15/2023, 8:42 PM
I remember trying this and it seemed to only apply to the user-code deployment pod, but not launched dagster-run pods. Let me give this a quick try though
d

daniel

02/15/2023, 8:43 PM
What version of dagster are you on currently?
The "includeConfigInLaunchedRuns" flag mentioned there is relatively new
ideally you'd be able to separately list the resources for the server and any launched runs, but that isn't quite there yet
❤️ 1
f

fahad

02/15/2023, 8:44 PM
ooo ok I missed that flag. Looks like the user-code deployment is on the latest release of dagster
d

daniel

02/15/2023, 8:46 PM
I think if your helm chart is past 0.15.0 or so then that behavior should be enabled by default
f

fahad

02/15/2023, 8:46 PM
The daemon and dagit are deployed separately from our user-code deployments so need to confirm versions there - not sure if that would have an impact
ok that worked for the time being!
ideally you’d be able to separately list the resources for the server and any launched runs, but that isn’t quite there yet
Would definitely be really neat to be able to specify this within the
Definitions
or something! But thanks for the help!!
d

daniel

02/15/2023, 9:01 PM
you mean just for the runs? There's a bit of a chicken/egg problem if it applies to the code servers since you'd need to start the code server first
f

fahad

02/15/2023, 9:02 PM
Exactly, just for the runs so that the user code server doesn’t have to sit at the resources meant for dagster-run pods
But it would be nice to specify defaults for dagster-run/dagster-step pods somewhere that also encompasses asset materializations (i.e. not just
Job.tags
) like you said
4 Views