Hey folks has anyone run Dagster jobs across 500+ nodes I m

Question

Hey folks has anyone run Dagster jobs across 500+ nodes I m looking at moving from kops to EKS and am concerned that EKS managed control plane is not going to handle 500 nodes starting at the same time

Noah K · Answer

EKS control plane wouldn t worry me but you should talk to your AWS sales engineer to make sure your quotas set up to allow scaling up down that fast

Noah K · Answer

You may need to change your cluster autoscaler config too

Noah K · Answer

500 nodes coming online at once isn t actually much work for kube apiserver compared to the normal baseline nonsense it deals with slightly smiling face

Alex Remedios · Answer

thanks Noah I wouldn t assume it s too intense as long as EKS is able to create more replicas DNS is a more pressing issue