Anyone using Dagster for pipelines where each job is a container? That’s my typical use case using Airflow or Luigi for now. I’ve played a bit with using Dagster for controling jobs on ECS Fargate.

hey Andrew, we haven’t invested in ECS Fargate support yet, but first-class support for fully containerized execution in Dagster is certainly where we’re headed. Presently we’re focused on getting this up and running on k8s, but I anticipate that we’ll eventually expand that workstream to cover other container orchestration systems.

@nate That would be awesome! We’ve used kubernetes in cases where the client already uses it, but that’s really only tech companies. ECS has a much lower barrier to entry so for a pure data engineering backend, makes things easier. For containers in general, would the container tasks be independent of dagster or run dagster software as a worker?