https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alexander Whillas

07/14/2022, 4:55 AM
I'm getting confused about dynamic outputs. I have a CSV coming in and it has variable number of columns that i must perform the same processing job on. I guess i'm not doing dynamic output, i'm launching jobs from an Op?
🤖 1
Actually, they all hit the same API which is rate limited, so i want to run the jobs sequentially..., I guess I do use dynamic output and run each in a sub-graph...? 🤔
y

yuhan

07/14/2022, 5:22 AM
You can do an op factory if you know the columns before runtime. Otherwise, the number of columns is dynamic, you’ll need dynamic outputs to generate downstream steps at runtime.
a

Alexander Whillas

07/14/2022, 5:23 AM
yeah, they are dynamic so i guess DynamicOutput it is ! Thanks mate
y

yuhan

07/14/2022, 5:24 AM
You can run ops sequentially with dynamic outputs. To do so, you can set max_concurrent to be 1 in the run config:
Copy code
execution:
  config:
    multiprocess:
      max_concurrent: 1
a

Alexander Whillas

07/14/2022, 5:25 AM
ah, thanks!
dagsir 1