https://dagster.io/ logo
Title
c

Carlos Pega

01/19/2023, 5:20 PM
Hi there! we are doing a POC in our company to process image pipelines and we were wondering if there's a way in which we could dynamicaly set the resources limits for a kubernetes OP (cpu, memory, nodegroup, etc, and also the docker image tag). We want to do this by triggering a pipeline with
RunRequest
executed from a sensor reacting to an external event (event that has those values we want to set in runtime). Thanks a lot in advance!
d

daniel

01/19/2023, 8:28 PM
Hi Carlos, is this using the k8s_job_op?
c

Carlos Pega

01/19/2023, 9:09 PM
You mean k8s op executor? Then yes 😄
kind reminder in case you missed this one @daniel 🙂 thanks in advance!!
d

daniel

01/20/2023, 3:26 PM
Assuming you meant the k8s_job_executor - I don't think we currently allow you to set individual asset or op tags from a RunRequest which would needed for this to work. If this would be useful for you would you mind filing a feature request issue?
c

Carlos Pega

01/20/2023, 3:27 PM
sure! thanks! do you think it could be any possible workaround there?
d

daniel

01/20/2023, 3:31 PM
Do you need to do this at the individaul op level? You can set run-level tags from a RunRequest that change the resource limits of the run worker pod
c

Carlos Pega

01/20/2023, 3:33 PM
yes, the idea is to do it at the individual op level because in our image processing pipeline we do map / collect and the collect resources needed vary depending on the amount of frames (maps) for the image
regarding the docker tag images perhaps it's enough to set it how you are saying taking into account that all ops share the same tag. Am i right?
d

daniel

01/20/2023, 3:47 PM
Are these images running Dagster Python code?
the ops that you want to run i mean
c

Carlos Pega

01/20/2023, 3:48 PM
yeap, everything runs in python
d

daniel

01/20/2023, 3:49 PM
typically you wouldn't need to override the image then - is it different than the image that you'd be using in your user code deployment?
c

Carlos Pega

01/20/2023, 4:09 PM
i will add some info. Let's imagine that each execution is made through a kafka event in a specific topic. A sensor reacts to that and all ops for the job are run with the latest docker tag for that pipeline. For some reason perhaps someone needs to run an execution with a specific different docker tag by sending an override docker tag in the event message
or they can just do it through the dagit launchpad
d

daniel

01/20/2023, 4:12 PM
the problem is that other things might have changed since that docker image too besides just the bodies of the ops
1
for example, if an op is added or removed between docker tagsd
c

Carlos Pega

01/20/2023, 4:14 PM
you are right. We need to think this a little bit more. Perhaps we can deploy a custom variation of the deployment with the needed changes for that execution and just treat it as an exception