https://dagster.io/ logo
n

NawafSheikh

03/17/2021, 11:37 AM
Is there a possibility to create multiple list of same type of schedulers based on the users from the db, like creating dynamic schedulers on the runtime?
d

daniel

03/17/2021, 12:38 PM
Hi - we don’t have great support for creating schedules dynamically, but we do have the ability to return multiple runs from a schedule tick, and those runs could be chosen dynamically (so for example, you could have a single schedule that each day launches a run for each user in a database). Would that work for what you’re looking for?
n

NawafSheikh

03/17/2021, 12:47 PM
Our use case is too create similar kind of list of schedules based on each user having different execution time based on the timezone. That can only be done using dynamically creating schedules for that user when a new user is created. Plus each user could have there own time for the next tick, like some users could tweak a scheduler to 15 min and some to an hour.
d

daniel

03/17/2021, 1:27 PM
approximately how many users are you imagining?
n

NawafSheikh

03/17/2021, 1:28 PM
100+ to 500+
d

daniel

03/17/2021, 2:04 PM
Got it. This sounds possible but tricky in the current system. You can definitely have a repository with dynamically-defined schedules, the repository function just needs to return a list of ScheduleDefinitions (and those can be created dynamically - see https://docs.dagster.io/examples/dep_dsl for an example with pipelines, but schedules could work similarly). The tricky thing would be that there would need to be something that turns on the schedules once they're created (and when a user is removed, you'd probably also want to delete any leftover schedules that were previously turned on for them, or there will be some logspew when the scheduler tries to find them and they're missing).
n

NawafSheikh

03/17/2021, 2:48 PM
Yeah. But is it possibile on the run. Like when the daemon is running and a new user just signed up. How a new Schedule would be added to the running repo?
d

daniel

03/17/2021, 2:55 PM
You can add or remove schedules while the daemon is running without restarting the daemon. The scheduler reloads your code every minute or so and would pick up any newly added schedules then.
n

NawafSheikh

03/17/2021, 2:57 PM
Ok thats great. I guess that would work. And possibly with the combination of graphql api it could be possibile to turn off the schedule when user doesn't exists. Great. Thanks
Can I know somehow that what are the current schedulers of the repo?
d

daniel

03/17/2021, 3:35 PM
the graphql API has this - there's a "schedules" field on the Repository object (Similar to the "pipelines" example on https://docs.dagster.io/overview/graphql-api)
n

NawafSheikh

03/17/2021, 3:46 PM
Ok. Thats great. Thanks a lot