The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

I'm getting confused about dynamic outputs. I have a CSV coming in and it has variable number of columns that i must perform the same processing job on. I guess i'm not doing dynamic output, i'm launching jobs from an Op?

Actually, they all hit the same API which is rate limited, so i want to run the jobs sequentially..., I guess I do use dynamic output and run each in a sub-graph...? :thinking_face:

You can do an op factory if you know the columns before runtime. Otherwise, the number of columns is dynamic, you’ll need dynamic outputs to generate downstream steps at runtime.

yeah, they are dynamic so i guess DynamicOutput it is ! Thanks mate

You can run ops sequentially with dynamic outputs. To do so, you can set max_concurrent to be 1 in the run config:
```execution:
  config:
    multiprocess:
      max_concurrent: 1```