https://dagster.io/ logo
#announcements
Title
# announcements
r

Rubén Lopez Lozoya

03/11/2021, 12:05 PM
Hey, I am using minikube to deploy a local cluster and trying to use the Helm chart provided with my values.yaml for my custom userDeployments. I am directly using my locally built docker image by setting pullPolicty to "never". The problem is that env variables passed to the userDeployments are not reaching the "run" pods, and when I attempt to run a pipeline it crashes since the env var cannot be retrieved. Is there any extra step I need to do to make those env variables visible to my deployment due to my configuration?
The dagster-release-maquinillo pod has the environment variables, but the run ones don't, do I need to provide these env variables somewhere else?
j

johann

03/11/2021, 12:54 PM
Hi @Rubén Lopez Lozoya, one way to do this is to load the configmap we created for the userDeployment in your runconfig. If you run
kubectl get cm
, you should see one following the form
{{ template "dagster.fullname" $ -}}-{{- $deployment.name }}-user-env
. That can be referenced in the playground with
Copy code
execution:
  celery-k8s:
    env_config_maps: <config-map-name>
This is a bit cumbersome, feel free to submit a feature request in a GH issue! cc @rex in case there have been any improvements here that I’m not aware of
r

Rubén Lopez Lozoya

03/11/2021, 12:58 PM
Hi Johann, thanks for the swift answer 🙂 I am not really experienced with k8s or Helm at all, but I was wondering if this "problem" is due to me using the Helm chart or the same would happen if I manage to create my own K8s deployments
j

johann

03/11/2021, 1:16 PM
r

Rubén Lopez Lozoya

03/11/2021, 1:16 PM
yeah I removed it bc I just saw this, so basically I need celery executors for k8s
👍 1
Thanks a lot!
j

johann

03/11/2021, 1:17 PM
I was wondering if this “problem” is due to me using the Helm chart or the same would happen if I manage to create my own K8s deployments
Regardless of the deployment, you’ll need to specify to our executor that it needs to attach a configmap with your env
r

Rubén Lopez Lozoya

03/11/2021, 1:18 PM
Cool, I believe this is better done with presets to avoid people from having to specify it over and over right? Ill always be using the same configmap
j

johann

03/11/2021, 1:25 PM
Yes, that would be ideal
r

Rubén Lopez Lozoya

03/11/2021, 2:17 PM
Johann sorry to bother you again, but I am reading Davids thread and I see he is not using Celery. If I want to use for example https://docs.dagster.io/_apidocs/libraries/dagster_k8s, how can I specify the env vars to the launcher so that Kubernetes jobs receive them?
nvm I figured it out
👍 1
j

johann

03/11/2021, 2:20 PM
For the DagsterK8s run launcher, you could specify _env_config_maps_ in your dagster.yaml, along with the other config shown on your page
r

Rubén Lopez Lozoya

03/11/2021, 2:21 PM
yeah, I need to have this dagster.yml in the dockerfile of my user code right?
just like with the docker-compose example?
j

johann

03/11/2021, 2:21 PM
We recommend sharing the dagster.yaml file with all parts of the deployment
r

Rubén Lopez Lozoya

03/11/2021, 2:22 PM
Is there any link to the docs so I can know how to do that?
j

johann

03/11/2021, 2:22 PM
If using the helm chart, we attach it as a configmap to all deployments
All you would need is
Copy code
#values.yaml
runLauncher:
  k8sRunLauncher:
    envConfigMaps:
      - <your configmap>
There is a weird feature gap here though with celery- the k8sRunLauncher is going to use the same configmaps for every pipeline it launches, where as we were able to change it per run with celery. Feel free to submit an issue!
r

Rubén Lopez Lozoya

03/11/2021, 2:28 PM
Thanks again Johann, you are being so helpful 🙂 These are my first steps with dagster and k8s
j

johann

03/11/2021, 2:29 PM
We’ll be happy to hear any feedback on how we can make the process smoother! We have a lot of docs work in process currently that we hope will help
r

Rubén Lopez Lozoya

03/11/2021, 2:39 PM
In the envConfigMaps i need to specify a name and my configmap, since it expects an object, what "name" should I pick? and where do I need to use that name
j

johann

03/11/2021, 2:45 PM
Afaik you should only need to specify a string, the name of the configmap
Which would be whatever configmap you want to attach. If it’s the same as earlier, then the
{{ template "dagster.fullname" $ -}}-{{- $deployment.name }}-user-env
which has the same env vars as you set on your user env
r

Rubén Lopez Lozoya

03/11/2021, 2:48 PM
This is the output from helm show values dagster/dagster. I believe I need to add the following:
Copy code
runLauncher:
  config:
    k8sRunLauncher:
      envConfigMaps:
        - name: dagster-release-maquinillo-user-env
This would be for the env vars, and I need another configmap for dagster.yml as you said?
j

johann

03/11/2021, 2:49 PM
We handle the dagster.yaml configmap if you’re using our helm chart
r

Rubén Lopez Lozoya

03/11/2021, 2:49 PM
cool
so just the env vars
j

johann

03/11/2021, 2:49 PM
You’re right with the runLauncher config, that looks good
r

Rubén Lopez Lozoya

03/11/2021, 2:52 PM
Yeah it definitely works, it only took me 3 hours to figure it all out lmao
Thanks a lot johann
8 Views