19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Can we use green celery k8s executor with cloud? W...
# dagster-plusc
Charlie Bini
03/27/2023, 1:50 PMCan we use green celery k8s executor with cloud? Was looking into it yesterday, but I could only find documentation for the self-hosted version. The cloud helm chart doesn’t appear to have the config options referenced?
d
daniel
03/27/2023, 1:52 PMWe don’t currently support this in the cloud helm chart- I don’t think it would be a huge lift to add though. Are you primarily interested in it for global op-level concurrency limits? (And of course the celery workers would be hosted in your cluster)
c
Charlie Bini
03/27/2023, 1:53 PMYeah that’s really the motivation for using it
Charlie Bini
03/27/2023, 1:54 PMWith GKE Autopilot, you’re restricted to a max total number of resources per cluster, so it’s possible for Dagster to overload it with requests
Charlie Bini
03/27/2023, 1:55 PMThat causes it to try to scale up when it can’t and starts killing pods
d
daniel
03/27/2023, 1:56 PMGot it - there’s also been some discussion lately about supporting our own celery-less global op limits, we don’t have a specific timeline yet for when that would be ready though
c
Charlie Bini
03/27/2023, 2:37 PMCool, i think that would be preferable to having you to worry about celery as well, but let me know if that’s added to the cloud helm chart in the meantime
2 Views