Can you create dynamic pipelines or have a pipeline execute a pipeline? If I get a list of items from a service, and I then want to run each item in the list through it' own pipeline, ideally naming the pipeline to the item name, what is the best way to do that? I am currently using a single pipeline and a composite solid, calling that solid for each item in the list. Disadvantage to that is the pipeline then has a solid for each record and visually in DAGIT doesn't look great. It would be good to have them disassociated in a sense, where one pipelines instantiates a series of pipelines for each list item, then those pipelines get scheduled to execute.

@max can you have a pipeline call a pipeline factory? if so, that's likely the best way to do it.

yes, some people are doing this kind of thing

Do you know how to do it? Is it PipelineDefinition?

https://docs.dagster.io/_apidocs/pipeline#dagster.pipeline use the pipeline decorator plus a pattern like the one https://docs.dagster.io/overview/solids-pipelines/solid-factories here for solids

Thanks. I'll give that a run.