Hi 👋🏽 . We are using the hybrid cloud. I am wondering if there is anything similar to airflow’s ECS operators? This has been an effective solution for us in the past where teams are in charge of creating/updating ECR images and the job orchestrators (ie airflow) running the images using ECS (or aws batch). I see that e.g pyspark_emr_step_launcher has similar vocabulary (launch_step, wait etc). Hopefully there is a readymade solution already 🤞🏽

You can use the EcsRunLauncher to launch the whole run in ECS but it isn't quite like ECS operators in that it's not as easy to provide your own image for a given op. I don't think there's a ready-made way to just launch ECS jobs and wait that's built into Dagster right now, but here's a gist of what I use to do this (if you decide to use it you'll probably need to modify in a couple places). I've been meaning to clean it up and PR it into Dagster in case their team is interested in it

I am really hoping to avoid

EcsRunLauncher

. Seems rather blunt solution OTOH, your gist is lovely. Thanks. Pretty close to what I am thinking. Maybe this ought to be a feature 🙂

Curious to hear a bit more about why the run launcher abstraction feels too blunt in this case, but yes, a step launcher (or just managing everything within an op like the gist does or like k8s_job_op) would be what you’d want. We don’t currently have a readymade ECS implementation of either in part because both tend to be a bit of anti-patterns for the typical use of Dagster since the explicitly bind the logic to the compute environment (losing the ability to run the same code in dev locally and in prod remotely)

both tend to be a bit of anti-patterns for the typical use of Dagster since the explicitly bind the logic to the compute environment

ahh. That is fair re: ECSRunLauncher. I had not seen the issue you linked to. The primary doc made it sound like the launcher was written to run Dagster agents or tasks that agents itself may spawn. I could not find any example/reference using it in a dagster workflow