Hi team, I'm working with @Jim Nisivoccia and @Zack D. to get our app running on celery/k8s. We are close to having it working, and we are running into the issue in the attached screenshot. In our docker/celery version of the app we had to set the vhost = dagster in the environment. I don't see anything in the helm chart about the vhost or any secrets that are being setup for rabbit. Should we be creating a secret and starting rabbit with env vars, or is there a better way to set the vhost with helm?

Hi Keith - this is an OK place to post - bit slower than usual due to respond with dagster day on the horizon, apologies. I'll confess that i'm not a rabbitmq expert and I had to look up what a vhost was in the first place, but we use the rabbitmq helm chart in our helm chart, and i see some information in their helm chart documentation about how to see the vhost: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq#configure-the-default-uservhost

Thanks @daniel I appreciate the response, I'll take a look at their helm chart and see if we can add overrides to our values.yaml. We've really been enjoying all the flexibility dagster offers and we're looking forward to finishing our conversion from dagster/docker/celery to dagster/k8s/celery. I can imagine that the team is really busy with dagster day coming up, so thanks again for the help. If it causes issues with the broker url, I'll post back here and let you know.

I don't think this would be the same problem - we fixed that one by bringing the celery docker executor more in line with the celery k8s executor

Yes, i remember the problem was solved in the k8s version and you refactored it to be shared with the docker version. I'll post more details tomorrow

Hey Daniel, I've tried plugging in the rabbitmq configuration listed in the rabbit helm chart that you shared and it didn't work with either of the two configurations I've tried in the images attached to this message. After clearing the volume rabbit stores its data in, when I do a helm install and bring up the new rabbitmq instance the default vhost is still set to "/".

hm, we're rapidly approaching the boundaries of things I have expertise in, but the configuration you have there looks different from what I see in https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq#configure-the-default-uservhost - it looks like they are using a multi-line string, but you're using a dict

No worries

it still keeps the liveness probe even if you set livenessProbe to empty on the run launcher config?

Yes, I tried commenting out the entire liveness probe section and it was still spawning the worker containers with the default probe attached

This is on a recent dagster version?

We are running on 0.15.4 currently

Mind posting the exact text of the values.yaml that wasn't working? Curious why the code I posted earlier wouldn't kick in if it was empty

runLauncher:

type: CeleryK8sRunLauncher

config:

# This configuration will only be used if the K8sRunLauncher is selected

# k8sRunLauncher:

#   # Change with caution! If you're using a fixed tag for pipeline run images, changing the

#   # image pull policy to anything other than "Always" will use a cached/stale image, which is

#   # almost certainly not what you want.

#   imagePullPolicy: "Never"

#   envSecrets:

#     - name: dagster-secrets

celeryK8sRunLauncher:

# Change with caution! If you're using a fixed tag for pipeline run images, changing the

# image pull policy to anything other than "Always" will use a cached/stale image, which is

# almost certainly not what you want.

imagePullPolicy: "Never"

# # The Celery workers can be deployed with a fixed image (no user code included)

# image:

#   # When a tag is not supplied for a Dagster provided image,

#   # it will default as the Helm chart version.

#   repository: "<http://datamaxdev1.jfrog.io/datamax/dev/docker_datamax_etl_pipelines|datamaxdev1.jfrog.io/datamax/dev/docker_datamax_etl_pipelines>"

#   tag: ~

#   pullPolicy: Always

# Support overriding the name prefix of Celery worker pods

nameOverride: "celery-workers"

# Additional config options for Celery, applied to all queues.

# These can be overridden per-queue below.

# For available options, see:

# <https://docs.celeryq.dev/en/stable/userguide/configuration.html>

configSource: {}

# Additional Celery worker queues can be configured here. When overriding, be sure to

# provision a "dagster" worker queue, as this is the default queue used by Dagster.

#

# Optionally, labels and node selectors can be set on the Celery queue's workers.

# Specifying a queue's node selector will override any existing node selector defaults.

# configSource will be merged with the shared configSource above.

workerQueues:

- name: "dagster"

replicaCount: 2

labels: {}

nodeSelector: {}

configSource: {}

additionalCeleryArgs: []

- name: "data-sourcing"

replicaCount: 2

labels: {}

nodeSelector: {}

configSource: {}

additionalCeleryArgs: []

# Additional environment variables to set on the celery/job containers

# A Kubernetes ConfigMap will be created with these environment variables. See:

# <https://kubernetes.io/docs/concepts/configuration/configmap/>

#

# Example:

#

# env:

#   ENV_ONE: one

#   ENV_TWO: two

env: {}

# Additional environment variables can be retrieved and set from ConfigMaps. See:

# <https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables>

#

# Example:

#

# envConfigMaps:

#   - name: config-map

envConfigMaps: []

# Additional environment variables can be retrieved and set from Secrets. See:

# <https://kubernetes.io/docs/concepts/configuration/secret/#use-case-as-container-environment-variables>

#

# Example:

#

# envSecrets:

#   - name: secret

envSecrets:

- name: dagster-secrets

annotations: {}

# Sets a node selector as a default for all Celery queues.

#

# See:

# <https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector>

nodeSelector: {}

# Support affinity and tolerations for Celery pod assignment. See:

# <https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity>

# <https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/>

affinity: {}

tolerations: []

podSecurityContext: {}

securityContext: {}

# Specify resources.

# Example:

#

# resources:

#   limits:

#     cpu: 100m

#     memory: 128Mi

#   requests:

#     cpu: 100m

#     memory: 128Mi

resources: {}

`# If

livenessProbe

does not contain

exec

field, then we will default to using:`

# exec:

#   command:

#     - /bin/sh

#     - -c

#     - dagster-celery status -A <http://dagster_celery_k8s.app|dagster_celery_k8s.app> -y {{ $.Values.global.dagsterHome }}/celery-config.yaml | grep "${HOSTNAME}:.*OK"

livenessProbe:

initialDelaySeconds: 15

periodSeconds: 10

timeoutSeconds: 10

successThreshold: 1

failureThreshold: 3

exec:

command:

- /bin/sh

- -c

- ls

# Additional volumes that should be included in the Job's Pod. See:

# <https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volume-v1-core>

#

# Example:

#

# volumes:

#   - name: my-volume

#     configMap: my-config-map

volumes: []

# Additional volume mounts that should be included in the container in the Job's Pod. See:

# See: <https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volumemount-v1-core>

#

# Example:

#

# volumeMounts:

#   - name: test-volume

#     mountPath: /opt/dagster/test_folder

#     subPath: test_file.yaml

volumeMounts: []

# Additional labels that should be included in the Job's Pod. See:

# <https://kubernetes.io/docs/concepts/overview/working-with-objects/labels>

#

# Example:

# labels:

#   my_label_key: my_label_value

labels: {}

# Whether the launched Kubernetes Jobs and Pods should fail if the Dagster run fails.

failPodOnRunFailure: false

Ah I bet livenessProbe: ~ would work better than commenting it out - commenting it out would fall back to the default

Let me give that a shot and get back to you

and the liveness probe is that "dagster-celery status -A dagster_celery_k8s.app" command?

Yea it's the default one that is described in the comments

OK, we'll investigate what's going on there - seems like you have a workaround in the short-term though?

Thank you Daniel, I appreciate it. For the time being we should be alright just running the ls command in the liveness probe