Can you create dynamic pipelines or have a pipeline execute

Question

Can you create dynamic pipelines or have a pipeline execute a pipeline If I get a list of items from a service and I then want to run each item in the list through it own pipeline ideally naming the pipeline to the item name what is the best way to do that I am currently using a single pipeline and a composite solid calling that solid for each item in the list Disadvantage to that is the pipeline then has a solid for each record and visually in DAGIT doesn t look great It would be good to have them disassociated in a sense where one pipelines instantiates a series of pipelines for each list item then those pipelines get scheduled to execute

Leor · Answer

< max> can you have a pipeline call a pipeline factory if so that s likely the best way to do it

Leor · Answer

John I m tagging another member of the team to weigh in on a potential solution

max · Answer

yes some people are doing this kind of thing

max · Answer

the disadvantage is you end up with a lot of pipeline runs in the run history

max · Answer

we d love feedback on what this would look like in a more perfect world

John Helewa · Answer

Do you know how to do it Is it PipelineDefinition

Leor · Answer

use the pipeline decorator plus a pattern like the one here for solids

John Helewa · Answer

Thanks I ll give that a run

John Helewa · Answer

< max> < Leor> I ve been able to make dynamic pipelines from using the factory model Thanks for that I ask other questions in another thread That was about how to get execurte pipeline to show up in dagit as a run history