https://dagster.io/ logo
#ask-community
Title
# ask-community
w

Wellington Braga

02/20/2023, 7:38 PM
Could someone guide me how to create dynamic pipelines, similarly to Airflow, but in Dagster. I have a json file in the format:
Copy code
[
    {"client_id": 1, "client_name": Joe},
    {"client_id": 2, "client_name": Ruth}
]
Where do I need to create a pipeline/job for each element in the list, with the same internal operations in the assets, only differentiating the id parameter. I searched in some forums, but little is said about dynamic jobs in Dagster. 😞 My question would be how to create a function in Dagster with operations similar to the
generate_compact_dags
function, defined in the
dynamic_pipelines.py file
dagster bot responded by community 2
a

Adam Bloom

02/20/2023, 8:03 PM
Asset/job/op factories are you friend here. The only example of this in the docs is under ops, but you can use factory patterns with most dagster concepts. https://docs.dagster.io/concepts/ops-jobs-graphs/ops#op-factory For your example, you’ll want to read your json file, and then create a list of ops/jobs/assets/etc using factory functions.
❤️ 2
w

Wellington Braga

02/20/2023, 8:32 PM
right when I started to implement dagster I came across Op Factory and I couldn't replicate it with my conditionals. I'll try to understand the functionality a little more. Thank you very much!
a

Adam Bloom

02/20/2023, 8:34 PM
Almost all of my pipelines are implemented with some sort of factory pattern - it’s super flexible.
e

Expoze Dummy

02/21/2023, 2:05 AM
Just implemented something like this last night and I didn't realize that I did an Op factory pattern. But yeah, this is the way
2 Views