Can you create dynamic pipelines or have a pipeline execute a pipeline? If I get a list of items from a service, and I then want to run each item in the list through it' own pipeline, ideally naming the pipeline to the item name, what is the best way to do that? I am currently using a single pipeline and a composite solid, calling that solid for each item in the list. Disadvantage to that is the pipeline then has a solid for each record and visually in DAGIT doesn't look great. It would be good to have them disassociated in a sense, where one pipelines instantiates a series of pipelines for each list item, then those pipelines get scheduled to execute.
07/09/2020, 4:26 PM
@max can you have a pipeline call a pipeline factory? if so, that's likely the best way to do it.
(John, I'm tagging another member of the team to weigh in on a potential solution)
07/09/2020, 5:02 PM
yes, some people are doing this kind of thing
the disadvantage is you end up with a lot of pipeline runs in the run history
we'd love feedback on what this would look like in a more perfect world
07/09/2020, 5:07 PM
Do you know how to do it? Is it PipelineDefinition?
@max@Leor I've been able to make dynamic pipelines from using the factory model. Thanks for that. I ask other questions in another thread. That was about how to get execurte_pipeline to show up in dagit as a run history.