Hi everyone, I drafted this design sketch to help better illustrated my question. This is a standard deployment from Dagster docs. What I want to add is to allow business people to modify the job configuration on demand. The modifications can be adding new jobs, deleting a job, turn off schedule for a job, set new schedule for a job, change some parameters for a job. As I understood, the jobs are registered by exposing @repository to dagit and dagster-daemon. In my case, for initial startup, I suppose I can load the configs from database to construct the jobs programatically. I am not sure how to handle update from this point. Note that we are not changing any user code. My question is, how can I reflect my on-demand config changes to Dagster? https://www.figma.com/file/5s1DJLQatpF0TxG0DOCOyI/Dagster-arch?node-id=0%3A1

Hi Kha - If you have a way to restart the user code server periodically (either when you know changes have been made or on some regular schedule), dagit will automatically reload your code, pick up any changes, and include them in the UI and in new runs

Hi Daniel, thank you for the reply. So the key is to restart the user code server.

That's right - that won't disrupt any ongoing runs if you're using one of the run launchers that launches each run in its own container (like docker / k8s / ecs)

If I have about 3000+ jobs (and the number can grow more), would this still be realistic?

That does sound like a bit more than average (might be a lot to manage all the different jobs in the dagit UI), but we'd be happy to talk through any scaling challenges that come up

As a consultant gig?

sorry, I don't totally follow

Ah I see.

sounds good! one thing to consider is that if the only difference between the jobs is the config, they don't necessarily all have to be different jobs - jobs can specify default config at definition time, but you can also specify config at runtime in the launchpad or over the graphql API.

I run these 3000+ jobs on cron schedule

I see - a job can have multiple schedules, and the schedule can supply the config, so that could work well

Okay, I will check the API

Schedules are loaded from code, just like jobs

Would a call to reload repository rerun the code defined in @repository block?

there's an API for turning schedules on and off, and you can set the default status of a schedule to be running as soon as its loaded

I see. Thanks! 😊