New to Dagster and hoping to get some clarity on running jobs in ECS. I don't see a "ecs_job_executor"... Does this mean it is not possible to horizontally scale jobs (via ecs) on ecs runs (e.g. execute parallel ops across ecs tasks). Therefore, scaling would need to be within a run via `multiprocess_executor`or we could use the dask_executor to scale horizontally? Thanks!

Correct. You can use the

EcsRunLauncher

to parallelize runs in their own ECS tasks. But there’s currently no

EcsExecutor

to parallelize steps within a run in their own ECS tasks. But yes, you should be able to use most of the other executors (probably all but the k8s ones and the docker one?) in conjunction with the

EcsRunLauncher

or you can implement your own

EcsExecutor

from the

Executor

abstract base class.

We're definitely planning on adding an ECS executor at some point, so let us know if that's a blocker for you and we can prioritize accordingly. I think one reason it hasn't come up yet is that ECS task spinup can be somewhat slow, so taking that on for each op could impose additional latency

Thanks @daniel and @jordan - that info is really helpful. We have both io bound and cpu intensive workloads, so trying to figure out the best, most flexible solution to supporting all of our pipelines/jobs. Is the dagster-dask executor reliable/stable?

I think we have more users of the celery executor than the dask executor to support the task queue use case for ops, but we have users using both for real workloads in production.