Hello All,
We have a use-case to create a pipeline dynamically from UI or API reference without the restart of dagster server? Is this possible? (newbie to dagster).
Like a button on dagit UI , we pass the required configuration via
API
and create pipeline on spot.
d
daniel
06/16/2021, 1:56 PM
Hi - the best way we have to support this at the moment is to make a pipeline that sources things dynamically from a YAML file (or database, or any other external source that you want) and have your API write to that external source. Here's an example that illustrates that: https://docs.dagster.io/concepts/solids-pipelines/pipelines#pipeline-dsl
You currently need to build that external source yourself though
Then you can reload that pipeline within dagit (without needing to restart dagit)