https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
k

Ken

08/07/2020, 4:20 PM
Hey Guys, Thank you again for the
K8S_RESOURCE_REQUIREMENTS_KEY
support implemented in: https://dagster.phacility.com/D3589 For triggering a full pipeline end-to-end, the resource requirements config work. But I see that if we trigger a subset of solids within that pipeline, the resource requirements is not applied. Is there any plan to support providing resource-requirement config on to GKE-Job runs for pipelines with solidSelection ?
Reproduction
s

schrockn

08/07/2020, 4:30 PM
Seems like a pretty clear bug! Do you mind filing an issue?
k

Ken

08/07/2020, 4:30 PM
Will do!
👍 1
Posted at: https://github.com/dagster-io/dagster/issues/2792 Thank you so much for looking into this!
j

johann

08/10/2020, 2:08 PM
Hi @Ken, trying to get a local reproduction. What dagster version are you running?
k

Ken

08/10/2020, 3:45 PM
Hello Johann, In that, I was running 0.8.6.
👍 1
j

johann

08/11/2020, 8:22 PM
Haven’t been able to reproduce this unfortunately, is it showing up on 0.9.1?
k

Ken

08/11/2020, 8:40 PM
Hello Johann, It is still showing to me in 0.9.1: • In the playground, if I set solid-selection to "*", resource-allocation tag is applied. • If solid-selection is set to a sub-selection => resource-allocation is not applied. Using the GKE ( Kubernetes ) => job yaml config, I confirm that resource-allocation is applied for the "*" config, and not applied for the sub-selection config. Could this be due to having solid-selection config will create a new pipeline, and that new pipeline does not inherit the pipeline-tags from the source pipeline ? Tell me if you need any more information to debug! ( Note: don't worry about the pipeline failure in that image, it is intentional )
j

johann

08/11/2020, 8:41 PM
Got it, thanks for the info
metal 1
y

yuhan

08/11/2020, 8:52 PM
hi @Ken what do you mean by “If solid-selection is set to a sub-selection”?
ah did you mean setting the
solid-selection
to anything other than
*
k

Ken

08/11/2020, 8:54 PM
Hello Yuhan, By that, I meant setting the field "solid_selection" into something other than "*", in other words, ask the run to process specified list of solids, and not all. You are correct 😄
y

yuhan

08/11/2020, 8:55 PM
kk thanks! trying to repo it on my end
j

johann

08/12/2020, 1:45 PM
@Ken a fix is in master and will be in the 0.9.2 release tomorrow. Thanks for catching this!
k

Ken

08/12/2020, 3:13 PM
Thank you Johann and everyone 😄
2 Views