George Pearse

12/09/2021, 3:03 PM
I often have non-technical users that want to define when to run a pipeline. Does anyone have a good design pattern for achieving this in Dagster e.g. via the yaml configs. Have a pipeline named add_scheduled_report, with a yaml config that allows you to define the Client Name, start date, and frequency.


12/09/2021, 7:02 PM
I think there are a couple of options here: 1. Create a schedule factory that reads yaml and dynamically constructs a schedule to include in the repository. One downside with this approach is that you would still have to start each schedule either through Dagit or through the CLI. 2. Create a custom sensor that implements its own logic to loop through each “virtual” schedule and kicks off runs (essentially acting as a dynamic scheduler). Downside of this is you would only have a single logical place to debug all of your ticks, instead of it being split up by report.