https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
m

mrdavidlaing

01/17/2021, 10:42 PM
I'm guessing; but it looks like celery worker_concurrency defaults to the #CPU cores on the K8s node. Fiddling with
config_source.worker_concurrency
as below I was able to get a single Celery worker to run multiple solids concurrently on a 1 VCPU GKE node. (Not sure this is actually makes things faster; but it does seem to "work")
Copy code
runLauncher:
  type: CeleryK8sRunLauncher
  config:
    celeryK8sRunLauncher:
      configSource:
        worker_concurrency: 4