Might be a weird question but… What determines how fast the QueuedRunCoordinator can launch runs? I’ve got a couple thousand small runs sitting in queue. They seem to complete faster than the coordinator launches new ones so I only have 2-4 in progress at a time.

hi Tyler - not a weird question! There's a dequeue_interval_seconds parameter that you can set on your dagster.yaml that defaults to 30 - that controls how frequently it checks to pull things off the queue. something like

module: dagster.core.run_coordinator
class: QueuedRunCoordinator
config:
  dequeue_interval_seconds: 5

This seems like it would be more of a throughput issue, if the thousands of runs are sticking around in the queue for a while

Thanks for the input @johann and @daniel! cpu/mem usage was sitting around 50-60%. I ended up with about 14k runs that had to process and they finished 7 hours later that night. We’ve currently got an Azure VM running our Dagster instance. I’m just using the DefaultRunLauncher. I am thinking I should be looking into something that will scale better in the long run. This was an initial extract for a new integration so it’s not our typical amount of runs. I feel kubernetes might of handled this better by scaling out. What’s your thoughts?

Did the number of in progress runs ever get close to your concurrency limit?

Just some context… I have a sensor kicking off a run to pull a file over SFTP to Azure blob storage. That’s the initial run. I also have a sensor kicking off a run to pull a file from blob, parse it and push it into Snowflake 🙂 The sftp -> blob run is extremely quick (1-2 sec) so they were just finishing faster than the launcher could launch new ones lol. Once it got to the second pipeline, it floated to my concurrency limit since they take a little longer.

Got it. Yeah having a high number of short pipelines is actually a dimension of stress on the system that we’re not as familiar with, compared to a lower number of large pipelines. We’re definitely interested in feedback. At some point the queue daemon will need to be able to scale horizontally to avoid being the bottleneck. It’s possible you ran in to that with your first set of runs, though that may also be solvable with a different launcher like you suggested

Thanks again with all the information! I’ll definitely look into using a different launcher and see what that looks like 🙂

Daniel and I chatted about this, and his initial suggestion that I dismissed might actually be a solution haha

I'm in a very similar predicament, so if anything's been improved in Dagster since (with respect to many short jobs in the queue), I'd be happy to learn about it. I have 2-10K runs in the queue, with these jobs usually taking ~10s to complete once scheduled to a pod by K8s. I've configured room for 512 concurrent runs in the QueuedRunCoordinator, however, I only see ~60-80 runs showing up in the in-progress tab in Dagit. So far I've: 1. Decreased

minimum_interval_seconds

on the sensor to 1s. Can it go to zero? 2. Provided 4X more CPU and memory to the daemon. 3. Scaled up my Dagster DB (initially there was some CPU pressure there, but that's been resolved). 4. Instrumented the latency for IO-sensitive parts of the sensor eval fn, as well as total tick latency. I'm seeing p99 total latency usually under 500ms, with some occasional spikes going to ~5s. 5. I periodically garbage-collect stale pods from the k8s API

I think there are still some improvements we need to make in both the sensor daemon and the run queue daemon to optimize them for throughput / reduce latency when there are a bunch of runs

Any open issues we can follow/contribute?

issue is here: https://github.com/dagster-io/dagster/issues/7763

Looks like we got it in 0.15.3, though I can't find any documentation of the new config. The source code indicates there are new

use_threads

and

num_workers

fields, but it's not immediately obvious where those should be set.

Hi @assaf, that will be configured in your

dagster.yaml

:

run_coordinator:
  ...
sensors:
  use_threads: true
  num_workers: 10

Or if using the helm chart, it will be configured in your `values.yaml`: https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L1006-L1011

Created issue at: https://github.com/dagster-io/dagster/issues/8737

I think there is docs/helm values but that PR unfortunately did not make it into the release. We’ll make sure this all gets updated for

0.15.4

. The relevant PR is here: https://github.com/dagster-io/dagster/pull/8657