https://dagster.io/ logo
Join Slack
Powered by
I would like to inject an environment variable int...
# deployment-kubernetes
t
I would like to inject an environment variable into my job. Is there an example I can look at to do this? I’ve tried
Copy code
dbt_job = my_dbt_job.to_job(resource_defs={"dbt": my_dbt_resource}, executor_def=k8s_job_executor, name="dbt_job")
and setting in Launchpad
Copy code
execution:
  config:
    env_vars: ["DAGSTER_DBT_EXAMPLE_PGHOST=dbt-example-postgresql.dev-fow-dagster"]
but I’m not seeing this env var in the job pod that launches.
j
Hi Tim, which K8s Job are you looking at? Would you be able to share the result of 
kubectl describe job <job name>
I think you’ll want to add that env var to the config for the K8sRunLauncher in your 
values.yaml
https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L431-L437
The config for the executor gets applied to the K8s Jobs that are launched for each op, while the run launcher config applies to those and the run worker Job which manages the whole run.
t
Copy code
❯ k describe job dagster-job-51dce79ce05bdbcff23a1d9dc176599e                                                                                                                                5.7s  Fri Feb 25 15:44:42 2022
Name:           dagster-job-51dce79ce05bdbcff23a1d9dc176599e
Namespace:      dev-fow-dagster
Selector:       controller-uid=12bcd946-710b-4909-a24d-c883e1d7079e
Labels:         <http://app.kubernetes.io/component=step_worker|app.kubernetes.io/component=step_worker>
                <http://app.kubernetes.io/instance=dagster|app.kubernetes.io/instance=dagster>
                <http://app.kubernetes.io/name=dagster|app.kubernetes.io/name=dagster>
                <http://app.kubernetes.io/part-of=dagster|app.kubernetes.io/part-of=dagster>
                <http://app.kubernetes.io/version=0.14.1|app.kubernetes.io/version=0.14.1>
                dagster/job=dbt_job
                dagster/op=dbt_seed_op
Annotations:    <none>
Parallelism:    1
Completions:    1
Start Time:     Thu, 24 Feb 2022 17:34:31 -0800
Completed At:   Thu, 24 Feb 2022 17:35:04 -0800
Duration:       33s
Pods Statuses:  0 Active / 1 Succeeded / 0 Failed
Pod Template:
  Labels:           <http://app.kubernetes.io/component=step_worker|app.kubernetes.io/component=step_worker>
                    <http://app.kubernetes.io/instance=dagster|app.kubernetes.io/instance=dagster>
                    <http://app.kubernetes.io/name=dagster|app.kubernetes.io/name=dagster>
                    <http://app.kubernetes.io/part-of=dagster|app.kubernetes.io/part-of=dagster>
                    <http://app.kubernetes.io/version=0.14.1|app.kubernetes.io/version=0.14.1>
                    controller-uid=12bcd946-710b-4909-a24d-c883e1d7079e
                    dagster/job=dbt_job
                    dagster/op=dbt_seed_op
                    job-name=dagster-job-51dce79ce05bdbcff23a1d9dc176599e
  Service Account:  dagster
  Containers:
   dagster:
    Image:      <http://artifactory.flexport.io/flexport-docker-sandbox-local/fow/dagster/ae:0.1.6-tchan|artifactory.flexport.io/flexport-docker-sandbox-local/fow/dagster/ae:0.1.6-tchan>
    Port:       <none>
    Host Port:  <none>
    Args:
      dagster
      api
      execute_step
      {"__class__": "ExecuteStepArgs", "instance_ref": {"__class__": "InstanceRef", "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name": "NoOpComputeLogManager", "config_yaml": "{}\n", "module_name": "dagster.core.storage.noop_compute_log_manager"}, "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData", "class_name": "PostgresEventLogStorage", "config_yaml": "postgres_db:\n  db_name: test\n  hostname: dagster-postgresql\n  params: {}\n  password:\n    env: DAGSTER_PG_PASSWORD\n  port: 5432\n  username: test\n", "module_name": "dagster_postgres.event_log"}, "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name": "LocalArtifactStorage", "config_yaml": "base_dir: /opt/dagster/dagster_home\n", "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__": "ConfigurableClassData", "class_name": "DefaultRunCoordinator", "config_yaml": "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data": {"__class__": "ConfigurableClassData", "class_name": "K8sRunLauncher", "config_yaml": "dagster_home:\n  env: DAGSTER_HOME\nenv_config_maps:\n- env: DAGSTER_K8S_PIPELINE_RUN_ENV_CONFIGMAP\nimage_pull_policy: Always\ninstance_config_map:\n  env: DAGSTER_K8S_INSTANCE_CONFIG_MAP\njob_namespace: dev-fow-dagster\nload_incluster_config: true\npostgres_password_secret:\n  env: DAGSTER_K8S_PG_PASSWORD_SECRET\nservice_account_name: dagster\n", "module_name": "dagster_k8s"}, "run_storage_data": {"__class__": "ConfigurableClassData", "class_name": "PostgresRunStorage", "config_yaml": "postgres_db:\n  db_name: test\n  hostname: dagster-postgresql\n  params: {}\n  password:\n    env: DAGSTER_PG_PASSWORD\n  port: 5432\n  username: test\n", "module_name": "dagster_postgres.run_storage"}, "schedule_storage_data": {"__class__": "ConfigurableClassData", "class_name": "PostgresScheduleStorage", "config_yaml": "postgres_db:\n  db_name: test\n  hostname: dagster-postgresql\n  params: {}\n  password:\n    env: DAGSTER_PG_PASSWORD\n  port: 5432\n  username: test\n", "module_name": "dagster_postgres.schedule_storage"}, "scheduler_data": {"__class__": "ConfigurableClassData", "class_name": "DagsterDaemonScheduler", "config_yaml": "{}\n", "module_name": "dagster.core.scheduler"}, "settings": {}}, "known_state": {"__class__": "KnownExecutionState", "dynamic_mappings": {}, "previous_retry_attempts": {}, "step_output_versions": []}, "pipeline_origin": {"__class__": "PipelinePythonOrigin", "pipeline_name": "dbt_job", "repository_origin": {"__class__": "RepositoryPythonOrigin", "code_pointer": {"__class__": "PackageCodePointer", "attribute": "analytics_engineering", "module": "fp.fow.ae.repository", "working_directory": "/app"}, "container_image": "<http://artifactory.flexport.io/flexport-docker-sandbox-local/fow/dagster/ae:0.1.6-tchan|artifactory.flexport.io/flexport-docker-sandbox-local/fow/dagster/ae:0.1.6-tchan>", "entry_point": ["dagster"], "executable_path": "/usr/local/bin/python"}}, "pipeline_run_id": "cb91c81a-7349-4cc7-873e-d9e82f33dba3", "retry_mode": {"__enum__": "RetryMode.DEFERRED"}, "should_verify_step": true, "step_keys_to_execute": ["dbt_seed_op"]}
    Environment Variables from:
      dagster-pipeline-env  ConfigMap  Optional: false
    Environment:
      DAGSTER_HOME:         /opt/dagster/dagster_home
      DAGSTER_PG_PASSWORD:  <set to the key 'postgresql-password' in secret 'dagster-postgresql-secret'>  Optional: false
    Mounts:
      /opt/dagster/dagster_home/dagster.yaml from dagster-instance (rw,path="dagster.yaml")
  Volumes:
   dagster-instance:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      dagster-instance
    Optional:  false
Events:        <none>
this is my 
values.yml
Copy code
dagit:
  env:
    DAGSTER_DBT_EXAMPLE_PGHOST: dbt-example-postgresql.dev-fow-dagster

  workspace:
    enabled: true
    servers:
      - host: analytics-engineering-repository.dev-fow-dagster
        port: 3030
        name: analytics-engineering
      - host: data-science-repository.dev-fow-dagster
        port: 3030
        name: data-science
      - host: analytics-engineering-helm.dev-fow-dagster
        port: 3030
        name: analytics-engineering-helm
      - host: k8s-example-user-code-1.dev-fow-dagster
        port: 3030
        name: k8s-example-user-code-1

dagster-user-deployments:
  enableSubchart: false

runLauncher:
  config:
    k8sRunLauncher:
      envVars:
        - DAGSTER_DBT_EXAMPLE_PGHOST
we would like teams to manage their own repository deployments, env vars for jobs, etc. what would be the best way to do this?
i was hoping to also be able add env vars to the job using Launchpad in dagit, which is why i posted the second snippet.
i think i understand now. we can only add env vars to the job via secrets or config maps, is that correct?
how does this line work https://github.com/dagster-io/dagster/blob/ec070e18885031b88c07862b48051a5fd75dba3[…]/libraries/dagster-k8s/dagster_k8s_tests/unit_tests/test_job.py i’ve set an env var in my user repo deployment and in dagit, but i don’t see it in the job.
j
Hmm, so
Copy code
k8sRunLauncher:
      envVars:
        - DAGSTER_DBT_EXAMPLE_PGHOST
will source the env from wherever the run is launching. That can be Dagit, or it can be the Daemon (if the QueuedRunCoordinator is enabled, or if the run is from a schedule or sensor). In general it is easier to put things in a configmap so you don’t have to set the env value in multiple places
👍 1
14 Views
Open in Slack
PreviousNext
Powered by