Hey folks, has anyone run Dagster jobs across 500+ nodes?
I’m looking at moving from kops to EKS and am concerned that EKS’ managed control plane is not going to handle 500 nodes starting at the same time.
10/13/2021, 9:25 PM
EKS control plane wouldn't worry me, but you should talk to your AWS sales engineer to make sure your quotas set up to allow scaling up/down that fast.
You may need to change your cluster-autoscaler config too
500 nodes coming online at once isn't actually much work for kube-apiserver compared to the normal baseline nonsense it deals with 🙂
10/14/2021, 8:19 AM
thanks Noah. I wouldn’t assume it’s too intense as long as EKS is able to create more replicas.
DNS is a more pressing issue.