Hi! Was wondering, what are some of your thoughts regarding running Dagster against ECS vs EKS? We're testing out right now and would love to hear other people's experiences.

At least as of 2022 Sep, Dagster didn't have an ECS execution mode that runs each op in its own container. We have a pipeline that we want to scale horizontally beyond one EC2's compute concurrently, so that meant k8s. I have run into some EKS limitations that surprised me, though. With the step-per-pod execution model, we were generating thousands of pods per day, and that seems to stress etcdserver and maybe other parts of the k8s control plane. AWS support wasn't much help and EKS doesn't provide much control plane visibility (so for example we couldn't directly tune etcdserver).

Thanks @Mark Fickett - really helpful! I recalled there should be a method of restricting the execution on Dagster to only a single pod per job to limit the number of pods spawned