I would like to inject an environment variable into my job I

Question

I would like to inject an environment variable into my job Is there an example I can look at to do this I ve tried ```dbt job = my dbt job to job resource defs= dbt my dbt resource executor def=k8s job executor name= dbt job ``` and setting in Launchpad ```execution config env vars DAGSTER DBT EXAMPLE PGHOST=dbt example postgresql dev fow dagster ``` but I m not seeing this env var in the job pod that launches

johann · Answer

Hi Tim which K8s Job are you looking at Would you be able to share the result of `kubectl describe job lt job name gt `

johann · Answer

I think you ll want to add that env var to the config for the K8sRunLauncher in your `values yaml` <https github com dagster io dagster blob master helm dagster values yaml L431 L437>

johann · Answer

The config for the executor gets applied to the K8s Jobs that are launched for each op while the run launcher config applies to those and the run worker Job which manages the whole run

Tim Chan · Answer

``` k describe job dagster job 51dce79ce05bdbcff23a1d9dc176599e 5 7s Fri Feb 25 15 44 42 2022 Name dagster job 51dce79ce05bdbcff23a1d9dc176599e Namespace dev fow dagster Selector controller uid=12bcd946 710b 4909 a24d c883e1d7079e Labels dagster job=dbt job dagster op=dbt seed op Annotations lt none gt Parallelism 1 Completions 1 Start Time Thu 24 Feb 2022 17 34 31 0800 Completed At Thu 24 Feb 2022 17 35 04 0800 Duration 33s Pods Statuses 0 Active 1 Succeeded 0 Failed Pod Template Labels controller uid=12bcd946 710b 4909 a24d c883e1d7079e dagster job=dbt job dagster op=dbt seed op job name=dagster job 51dce79ce05bdbcff23a1d9dc176599e Service Account dagster Containers dagster Image Port lt none gt Host Port lt none gt Args dagster api execute step class ExecuteStepArgs instance ref class InstanceRef compute logs data class ConfigurableClassData class name NoOpComputeLogManager config yaml n module name dagster core storage noop compute log manager custom instance class data null event storage data class ConfigurableClassData class name PostgresEventLogStorage config yaml postgres db n db name test n hostname dagster postgresql n params n password n env DAGSTER PG PASSWORD n port 5432 n username test n module name dagster postgres event log local artifact storage data class ConfigurableClassData class name LocalArtifactStorage config yaml base dir opt dagster dagster home n module name dagster core storage root run coordinator data class ConfigurableClassData class name DefaultRunCoordinator config yaml n module name dagster core run coordinator run launcher data class ConfigurableClassData class name K8sRunLauncher config yaml dagster home n env DAGSTER HOME nenv config maps n env DAGSTER K8S PIPELINE RUN ENV CONFIGMAP nimage pull policy Always ninstance config map n env DAGSTER K8S INSTANCE CONFIG MAP njob namespace dev fow dagster nload incluster config true npostgres password secret n env DAGSTER K8S PG PASSWORD SECRET nservice account name dagster n module name dagster k8s run storage data class ConfigurableClassData class name PostgresRunStorage config yaml postgres db n db name test n hostname dagster postgresql n params n password n env DAGSTER PG PASSWORD n port 5432 n username test n module name dagster postgres run storage schedule storage data class ConfigurableClassData class name PostgresScheduleStorage config yaml postgres db n db name test n hostname dagster postgresql n params n password n env DAGSTER PG PASSWORD n port 5432 n username test n module name dagster postgres schedule storage scheduler data class ConfigurableClassData class name DagsterDaemonScheduler config yaml n module name dagster core scheduler settings known state class KnownExecutionState dynamic mappings previous retry attempts step output versions pipeline origin class PipelinePythonOrigin pipeline name dbt job repository origin class RepositoryPythonOrigin code pointer class PackageCodePointer attribute analytics engineering module fp fow ae repository working directory app container image entry point dagster executable path usr local bin python pipeline run id cb91c81a 7349 4cc7 873e d9e82f33dba3 retry mode enum RetryMode DEFERRED should verify step true step keys to execute dbt seed op Environment Variables from dagster pipeline env ConfigMap Optional false Environment DAGSTER HOME opt dagster dagster home DAGSTER PG PASSWORD lt set to the key postgresql password in secret dagster postgresql secret gt Optional false Mounts opt dagster dagster home dagster yaml from dagster instance rw path= dagster yaml Volumes dagster instance Type ConfigMap a volume populated by a ConfigMap Name dagster instance Optional false Events lt none gt ```

Tim Chan · Answer

this is my `values yml` ```dagit env DAGSTER DBT EXAMPLE PGHOST dbt example postgresql dev fow dagster workspace enabled true servers host analytics engineering repository dev fow dagster port 3030 name analytics engineering host data science repository dev fow dagster port 3030 name data science host analytics engineering helm dev fow dagster port 3030 name analytics engineering helm host k8s example user code 1 dev fow dagster port 3030 name k8s example user code 1 dagster user deployments enableSubchart false runLauncher config k8sRunLauncher envVars DAGSTER DBT EXAMPLE PGHOST```

Tim Chan · Answer

we would like teams to manage their own repository deployments env vars for jobs etc what would be the best way to do this

Tim Chan · Answer

i was hoping to also be able add env vars to the job using Launchpad in dagit which is why i posted the second snippet

Tim Chan · Answer

i think i understand now we can only add env vars to the job via secrets or config maps is that correct

Tim Chan · Answer

how does this line work <https github com dagster io dagster blob ec070e18885031b88c07862b48051a5fd75dba39 python modules libraries dagster k8s dagster k8s tests unit tests test job py L168|https github com dagster io dagster blob ec070e18885031b88c07862b48051a5fd75dba3 libraries dagster k8s dagster k8s tests unit tests test job py> i ve set an env var in my user repo deployment and in dagit but i don t see it in the job

johann · Answer

Hmm so ```k8sRunLauncher envVars DAGSTER DBT EXAMPLE PGHOST``` will source the env from wherever the run is launching That can be Dagit or it can be the Daemon if the QueuedRunCoordinator is enabled or if the run is from a schedule or sensor In general it is easier to put things in a configmap so you don t have to set the env value in multiple places