I'm using

EcsRunLauncher

and my

QueuedRunCoordinator

is setup with

max_concurrent_runs

set to 750. A sensor requested ~650 job runs but I never saw more than about 200 in progress at once even though there were hundreds of runs waiting in queue. I expected to see a larger number of jobs in progress at once. The db I use for run and event storage as well as my grpc service did hit 100% CPU utilization at times. Any advice for getting runs out of the queue faster and have in progress runs closer to the

max_concurrent_runs

value?

Have you configured your instance to use threads in order to evaluate sensors and schedules? This should increase your throughput: https://docs.dagster.io/deployment/dagster-instance#sensor-evaluation

I think for parallelizing the runs coming off the queue you'd want something like this (similar to what rex posted, but for taking runs off the queue instead of putting them on):

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    dequeue_use_threads: true
    dequeue_num_workers: 8 # This number can be tuned depending on your parallelism needs

I enabled both of these and throughput definitely increased when enqueuing and dequeuing. I have run monitoring enabled

run_monitoring:
  enabled: true
  start_timeout_seconds: 300

and since enabling threading I'm seeing some failures due to a job exceeding the

start_timeout_seconds

. Is this a sign to scale up the daemon?

Do you have a sense of what was happening during those 300 seconds for the run that failed? What was the last line in the event log for that run before run monitoring killed it? I wouldn’t actually expect the daemon to be in the critical path there - but I’ve definitely seen ECS fargate sometimes take more than 300 seconds to spin up a task

image.png

What version of dagster are you using?

1.1.18

It's surprising that the task ARN wouldn't be in ECS, since the only place we would get such an ARN from would be the ECS API (it comes from the result of the run_task API call): https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/ecs/launcher.py#L394-L430 Are you sure the tasks aren't there but in a STOPPED state?