Hi, I was wondering about same thing and so tried to look for similar questions while trying to spin up ECS task as an op. Do we have any updates on “ecs_job_executor” yet? https://dagster.slack.com/archives/C014UDS8LAV/p1648523436201229

Hi Pragna - could you share more about your use case? One reason this hasn't been a huge priority so far is that ECS fargate tasks can take a long time to spin up - so incurring that cost once per run might be more preferable to incurring it for every individual op

How long is "a long time to spin up"? This article makes it sound like ECS+Fargate could spin up thousands of tasks in a few minutes. We're also hoping to scale horizontally; there are some setup ops in our job, but the bulk of it is running ~1k ops which can go in parallel, and take 1-20 minutes each. So 1m startup overhead would be fine. I guess I'd misunderstood something previously thinking Dagster would do an op per task on ECS+Fargate. In general I've been pretty confused by the cross product of job-v-op and executor, if there is now or if there could be a summary table of what's available in the docs that would be a big help. I had been leaning towards ECS instead of EKS since it has slightly simpler setup / lower overhead due to not having to run persistent clusters.

Hi Daniel, we are using it to run one of our models from ECR repository and also we are moving away from k8s to work more independently by using ECS tasks.

Got it - and Pragna just to confirm, you're aware that we do support running the whole job in ECS right? We just don't (yet) support running each individual op in an ECS task yet

Yes, I am aware about that. Thanks

Mark that graph looks very plausible to me - I'm sure we'll add an ECS executor and I could see it being useful in the case you mentioned (where you want to run lots of things in parallel and don't mind a fixed startup cost to launch those many things). I think the first thing we'll do in the ECS arena is add ECS on EC2 support though since there's been a bit more demand for that so far

Thanks Daniel. I didn't realize the Dagster executor would need to know what capacity provider ECS was using. Maybe my mental model of the layers here is wrong. I had pictured Dagster handing a task to ECS, and then ECS handing that task to whatever its compute was, be that EC2, EC2 autoscaling groups, or Fargate.

ECS requires you to tell it which capacity provider you're using - adding EC2 support might be pretty straightforward / as simple as changing what we tell it