https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
r

rowan gaffney

03/29/2022, 3:10 AM
New to Dagster and hoping to get some clarity on running jobs in ECS. I don't see a "ecs_job_executor"... Does this mean it is not possible to horizontally scale jobs (via ecs) on ecs runs (e.g. execute parallel ops across ecs tasks). Therefore, scaling would need to be within a run via `multiprocess_executor`or we could use the dask_executor to scale horizontally? Thanks!
j

jordan

03/29/2022, 2:23 PM
Correct. You can use the
EcsRunLauncher
to parallelize runs in their own ECS tasks. But there’s currently no
EcsExecutor
to parallelize steps within a run in their own ECS tasks. But yes, you should be able to use most of the other executors (probably all but the k8s ones and the docker one?) in conjunction with the
EcsRunLauncher
or you can implement your own
EcsExecutor
from the
Executor
abstract base class.
ty thankyou 1
d

daniel

03/29/2022, 2:27 PM
We're definitely planning on adding an ECS executor at some point, so let us know if that's a blocker for you and we can prioritize accordingly. I think one reason it hasn't come up yet is that ECS task spinup can be somewhat slow, so taking that on for each op could impose additional latency
r

rowan gaffney

03/29/2022, 2:58 PM
Thanks @daniel and @jordan - that info is really helpful. We have both io bound and cpu intensive workloads, so trying to figure out the best, most flexible solution to supporting all of our pipelines/jobs. Is the dagster-dask executor reliable/stable?
d

daniel

03/29/2022, 3:00 PM
I think we have more users of the celery executor than the dask executor to support the task queue use case for ops, but we have users using both for real workloads in production.
13 Views