Simon04/21/2022, 10:01 AM
The scheduler daemon is responsible for creating runs from any schedules that are turned on. This daemon will run as long as you have not overridden the defaultMy assumption is that the scheduler daemon creates runs that are in a queued state. Is that correct? My assumption for this is because the run coordinator seems to have an impact on which/how many runs actually get run, if I for example setas the scheduler on your instance.
to 0 none will actually run. And AFAIK the run coordinator acts on queued Runs.
But the docs don't mention any of this. Also on the run coordinator it only says here https://docs.dagster.io/deployment/overview#job-execution-flow
The Run Coordinator is a class invoked by the Dagit process when runs are launched from the Dagit UI. The run coordinator class can be configured to pass runs to the Daemon via a queue.The first part sounds like the run coordinator is only relevant when triggering runs via Dagit. The second part I don't really understand what it's trying to say. The first paragraph of this section does say
Runs launched by schedules and sensors go through the same flow, but the first step is called by the Dagster Daemon instead of Dagit.But this isn't reflected in the rest. The same for run coordinator page https://docs.dagster.io/deployment/run-coordinator
In production Dagster deployments, there are often many runs being launched at once. The run coordinator lets you control the policy that Dagster uses to manage the set of runs in your deployment. When you submit a run from Dagit or the Dagster command line, it is first sent to the run coordinator, which applies any limits or prioritization policies before eventually sending it to the run launcher to be launched.Again only Dagit (and the CLI) are mentioned, nothing about scheduled (or probably more correct? queued) runs is mentioned. The scheduler/scheduled runs are only mentioned in passing in the defaultruncoordinator section https://docs.dagster.io/deployment/run-coordinator#defaultruncoordinator Also as a more generic suggestion I think a picture describing this process would make it a lot clearer.