Hi team, can we pass dynamic environment variables to run launchers? For now, our run launcher is using the static environment variables we set in the dagster-user-deployment. For DEV/QA/PROD environments and incoming new environments, we would like to pass different sets of environment variables to each run launcher. We would like to pass environment name in run config. Before going into run launchers, can we run a python script for fetching secrets based on environment name in run_config and send new sets of the environment variables to run launchers? Below is our current values.yaml for user deployment and run launcher:

dagster-user-deployments:
	enabled: true
	imagePullSecrets:
	   - name: regcred
	deployments:
	  - name: "pipeline1"
		image:
		  repository: 
		  # repository: 
		  tag: 
		  pullPolicy: Always
		dagsterApiGrpcArgs:
		  - "-h"
		  - "0.0.0.0"
		  - "--working-directory"
		  - "/opt/dagster/home"
		  - "--python-file"
		  - "orchestration_manager/repositories/pipeline.py"
		port: 5060
		envSecrets:
		  - name: dagster-secrets
		includeConfigInLaunchedRuns:
		  enabled: true

runLauncher:
	type: K8sRunLauncher
	config:
	  k8sRunLauncher:
		imagePullPolicy: Always
		envSecrets:
		  - name: dagster-secrets

Would tags on jobs work for you? We're using https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment to set tags on certain jobs that add additional env from secrets - can share the syntax if useful

Hi @Adam Bloom, thanks for quick response. I would much appreciate if you can point out where to add the python script in dagster-k8s/config. Since we have lots of jobs, instead of adding secret fetching logic in the resources or ops used for a job, I would prefer adding the logics before kicking off runLaunchers to make the environment variables swapping on runLauncher level not on jobs level.

I'm personally surprised you use a single deployment, instead of a user code deployment per env. If its already the latter, then can't you just change your current deployment configuration? To add an env from to an individual job, we do something like this:

k8s_cfg["container_config"] = {"env_from": [{"secret_ref": {"name": <name>}}]}

and tag the job according to the docs linked above

Hey Zack - we don't support running arbitrary python before launching the run, that might require writing a custom run launcher of some sort (The run launcher is typically totally isolated from the code of the jobs being run). The tags that Adam mentioned can be used to insert specific environment variables into specific jobs, but that wouldn't take the run config into account. You can also configure environment variables and secrets on user code deployments and those will also be incorporated into any launched runs: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#configure-your-user-deployment - those don't take the run config into account or have the ability to run python scripts either though