Kenneth Cruz08/10/2021, 7:16 AM
alex08/10/2021, 2:53 PM
so the pipelines will have to be created dynamicallyCan you expand on this? Outside of specific dynamic map & collect features, dagster operates on a static pipeline structure once execution begins. We have examples of creating pipelines “dynamically” but they are locked in once execution starts.
does it make sense to just deploy Dagster as a python module to check and execute pipelines if there is no need for the dagityea that should work just fine
Kenneth Cruz08/10/2021, 10:20 PM
Can you expand on this?Thanks. As for dynamic pipeline creation, this is having to create a dynamic number branches out from certain solids, then be collected later. I guess another way to do it instead is just create all the branches beforehand and just yield the specific outputs depending on some conditions. They won't change once they are executed.
yea that should work just finecool. It just not able to utilise dagster's daemon functions but more of just using API's for pipeline and solid creation.
alex08/11/2021, 2:19 PM