https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
f

Felipe Saldana

02/12/2021, 11:58 PM
So I believe I have everything deployed correctly for local K8 + Celery but my pipeline is not actually running. Am I missing something that actually tells celery to "grab" the job and process?
Copy code
helm upgrade --install dagster-release --namespace dagster dagster/dagster -f values.yaml --set runLauncher.type=CeleryK8sRunLauncher \
--set dagsterDaemon.queuedRunCoordinator.enabled=true \
--set rabbitmq.enabled=true \
--set flower.enabled=true
n

nate

02/13/2021, 12:14 AM
hi Felipe, one thing I might check is whether that run coordinator job mentioned in your first screenshot launched successful and if there are any logs from the associated pod
plus1 1
j

johann

02/13/2021, 12:16 AM
The run worker pod name is logged in your first screenshot, the commands would be
Copy code
kubectl describe pod dagster-run-<ID>
kubectl logs dagster-run-<ID>
f

Felipe Saldana

02/15/2021, 5:11 PM
Thanks, using describe/logs allowed me to see part of my issue. I am only running this locally at the moment and needed to change the image_pull_policy and the job_namespace
Copy code
execution:
  celery-k8s:
    config:
      image_pull_policy: IfNotPresent
      job_namespace: dagster
j

johann

02/15/2021, 5:17 PM
Glad you were able to work this out, it’s working as expected now?
We also are tracking a few issues similar to what you ran into here where we don’t expose k8s error info in Dagit. Feel free to contribute any thoughts! https://github.com/dagster-io/dagster/issues/2856
f

Felipe Saldana

02/15/2021, 5:21 PM
Well, not exactly. I need to read through how _env_config_maps_ work. I am getting issues like below
Copy code
dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'instance_config_map': Field(<dagster.config.source.StringSourceType object at 0x7f97a3d385d0>, default=@, is_required=True), 'postgres_password_secret': Field ......
j

johann

02/15/2021, 5:56 PM
Is there more to the error message? Where is this error surfaced?
f

Felipe Saldana

02/15/2021, 6:35 PM
I was not pointing to a valid configmap. I pointed to a valid one and got celery to actually kick things off.
j

johann

02/15/2021, 7:01 PM
🎉
🎉 1