Could someone guide me how to create dynamic pipelines, similarly to Airflow, but in Dagster. I have a json file in the format:

[
    {"client_id": 1, "client_name": Joe},
    {"client_id": 2, "client_name": Ruth}
]

Where do I need to create a pipeline/job for each element in the list, with the same internal operations in the assets, only differentiating the id parameter. I searched in some forums, but little is said about dynamic jobs in Dagster. 😞 My question would be how to create a function in Dagster with operations similar to the

generate_compact_dags

function, defined in the

dynamic_pipelines.py file

Asset/job/op factories are you friend here. The only example of this in the docs is under ops, but you can use factory patterns with most dagster concepts. https://docs.dagster.io/concepts/ops-jobs-graphs/ops#op-factory For your example, you’ll want to read your json file, and then create a list of ops/jobs/assets/etc using factory functions.

right when I started to implement dagster I came across Op Factory and I couldn't replicate it with my conditionals. I'll try to understand the functionality a little more. Thank you very much!

Almost all of my pipelines are implemented with some sort of factory pattern - it’s super flexible.

Just implemented something like this last night and I didn't realize that I did an Op factory pattern. But yeah, this is the way