https://dagster.io/ logo
a

Andrew Madonna

03/04/2020, 7:24 PM
Anyone using Dagster for pipelines where each job is a container? That’s my typical use case using Airflow or Luigi for now. I’ve played a bit with using Dagster for controling jobs on ECS Fargate.
n

nate

03/05/2020, 11:00 AM
hey Andrew, we haven’t invested in ECS Fargate support yet, but first-class support for fully containerized execution in Dagster is certainly where we’re headed. Presently we’re focused on getting this up and running on k8s, but I anticipate that we’ll eventually expand that workstream to cover other container orchestration systems.
a

Andrew Madonna

03/05/2020, 7:16 PM
@nate That would be awesome! We’ve used kubernetes in cases where the client already uses it, but that’s really only tech companies. ECS has a much lower barrier to entry so for a pure data engineering backend, makes things easier. For containers in general, would the container tasks be independent of dagster or run dagster software as a worker?