I m using `EcsRunLauncher` and my `QueuedRunCoordinator` is

Question

I m using `EcsRunLauncher` and my `QueuedRunCoordinator` is setup with `max concurrent runs` set to 750 A sensor requested ~650 job runs but I never saw more than about 200 in progress at once even though there were hundreds of runs waiting in queue I expected to see a larger number of jobs in progress at once The db I use for run and event storage as well as my grpc service did hit 100 CPU utilization at times Any advice for getting runs out of the queue faster and have in progress runs closer to the `max concurrent runs` value

rex · Answer

Have you configured your instance to use threads in order to evaluate sensors and schedules This should increase your throughput <https docs dagster io deployment dagster instance sensor evaluation>

rex · Answer

Something like in your `dagster yaml` ```sensors use threads true num workers 8 schedules use threads true num workers 8```

daniel · Answer

I think for parallelizing the runs coming off the queue you d want something like this similar to what rex posted but for taking runs off the queue instead of putting them on ```run coordinator module dagster core run coordinator class QueuedRunCoordinator config dequeue use threads true dequeue num workers 8 This number can be tuned depending on your parallelism needs```

Tom Reilly · Answer

I enabled both of these and throughput definitely increased when enqueuing and dequeuing I have run monitoring enabled ```run monitoring enabled true start timeout seconds 300``` and since enabling threading I m seeing some failures due to a job exceeding the `start timeout seconds` Is this a sign to scale up the daemon

daniel · Answer

Do you have a sense of what was happening during those 300 seconds for the run that failed What was the last line in the event log for that run before run monitoring killed it I wouldn t actually expect the daemon to be in the critical path there but I ve definitely seen ECS fargate sometimes take more than 300 seconds to spin up a task

Tom Reilly · Answer

image png

Tom Reilly · Answer

The engine event logs out ` EcsRunLauncher Launching run in ECS task` with a task arn and then about 5min later the run failure is triggered

Tom Reilly · Answer

In ECS I do not see a task with the arn logged out by the `ENGINE EVENT`

Tom Reilly · Answer

for additional context I m load testing with a set of about 700 files The sensor runs a query that returns the 700 files and initiates a `RunRequest` for each

Tom Reilly · Answer

of those 700 since I enabled threading I m seeing about 2 5 fail each time due to exceeding `start timeout seconds`

Tom Reilly · Answer

Manually checking a dozen or so successful job runs the time between the `RUN STARTING` and `RUN START` is usually around 70 seconds

daniel · Answer

What version of dagster are you using

Tom Reilly · Answer

1 1 18

daniel · Answer

It s surprising that the task ARN wouldn t be in ECS since the only place we would get such an ARN from would be the ECS API it comes from the result of the run task API call <https github com dagster io dagster blob master python modules libraries dagster aws dagster aws ecs launcher py L394 L430> Are you sure the tasks aren t there but in a STOPPED state

Tom Reilly · Answer

Tom Reilly · Answer

looks the task expires after an hour

Tom Reilly · Answer

For now with sensor threading disabled and dequeue threading enabled I can consistently process the set of files without failures Will leave that disabled for now and if I encounter the error I ll make sure to check the task before it expires