Hey, I am using minikube to deploy a local cluster and trying to use the Helm chart provided with my values.yaml for my custom userDeployments. I am directly using my locally built docker image by setting pullPolicty to "never". The problem is that env variables passed to the userDeployments are not reaching the "run" pods, and when I attempt to run a pipeline it crashes since the env var cannot be retrieved. Is there any extra step I need to do to make those env variables visible to my deployment due to my configuration?

Hi @Rubén Lopez Lozoya, one way to do this is to load the configmap we created for the userDeployment in your runconfig. If you run

kubectl get cm

, you should see one following the form

{{ template "dagster.fullname" $ -}}-{{- $deployment.name }}-user-env

. That can be referenced in the playground with

execution:
  celery-k8s:
    env_config_maps: <config-map-name>

Hi Johann, thanks for the swift answer 🙂 I am not really experienced with k8s or Helm at all, but I was wondering if this "problem" is due to me using the Helm chart or the same would happen if I manage to create my own K8s deployments

Hi @Rubén Lopez Lozoya, https://docs.dagster.io/_apidocs/libraries/dagster_celery_k8s#dagster_celery_k8s.celery_k8s_job_executor should help with the prior question

yeah I removed it bc I just saw this, so basically I need celery executors for k8s

I was wondering if this “problem” is due to me using the Helm chart or the same would happen if I manage to create my own K8s deployments

Regardless of the deployment, you’ll need to specify to our executor that it needs to attach a configmap with your env

Cool, I believe this is better done with presets to avoid people from having to specify it over and over right? Ill always be using the same configmap

Yes, that would be ideal

Johann sorry to bother you again, but I am reading Davids thread and I see he is not using Celery. If I want to use for example https://docs.dagster.io/_apidocs/libraries/dagster_k8s, how can I specify the env vars to the launcher so that Kubernetes jobs receive them?

For the DagsterK8s run launcher, you could specify _env_config_maps_ in your dagster.yaml, along with the other config shown on your page

yeah, I need to have this dagster.yml in the dockerfile of my user code right?

We recommend sharing the dagster.yaml file with all parts of the deployment

Is there any link to the docs so I can know how to do that?

If using the helm chart, we attach it as a configmap to all deployments

Thanks again Johann, you are being so helpful 🙂 These are my first steps with dagster and k8s

We’ll be happy to hear any feedback on how we can make the process smoother! We have a lot of docs work in process currently that we hope will help

In the envConfigMaps i need to specify a name and my configmap, since it expects an object, what "name" should I pick? and where do I need to use that name

Afaik you should only need to specify a string, the name of the configmap

This is the output from helm show values dagster/dagster. I believe I need to add the following:

runLauncher:
  config:
    k8sRunLauncher:
      envConfigMaps:
        - name: dagster-release-maquinillo-user-env

This would be for the env vars, and I need another configmap for dagster.yml as you said?

We handle the dagster.yaml configmap if you’re using our helm chart

cool

You’re right with the runLauncher config, that looks good

Yeah it definitely works, it only took me 3 hours to figure it all out lmao