Hello, I'm trying to run multiple ops with Dynamic output. However I only want the result from Dynamic output to run after the previous result is complete. Is there any way to do it beside set
max_concurrent = 1
for the job as I still run other ops at the same time.
n
Nick Dellosa
06/08/2023, 4:22 PM
Do you mean kick off the dynamic ops as soon as they're inputs are yielded or run the op passed to a map in parallel? Or do you mean run the op passed to a map sequentially?
q
Quan Dao Hong
06/08/2023, 5:07 PM
I mean to run the map but config to run 1 job in a map at 1 time
Quan Dao Hong
06/08/2023, 5:07 PM
I'm thinking about using tag_concurrency_limits
s
sandy
06/08/2023, 11:22 PM
I don't believe that there's currently a way to do this. If you're up for filing a github issue, it would help us track this request.
Also, we're currently working on this issue: https://github.com/dagster-io/dagster/issues/12470. Which could help once it's addressed.