https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Andrew Maguire

08/02/2023, 4:03 PM
is there a way for me to have this be driven by an env var so its not hardcoded in
dagster.yaml
?
just wondering if some tricks or way to do similar in dagster.yaml so i dont have to hardcode it like above.
idea being once a user sets
GOOGLE_APPLICATION_CREDENTIALS
env var to point at creds file then they are good to go
i see this errror:
Copy code
docker_example_daemon      | 2023-08-02 16:08:05 +0000 - dagster.daemon.QueuedRunCoordinatorDaemon - ERROR - Caught an unrecoverable error while dequeuing the run. Marking the run as failed and dropping it from the queue: docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.43/containers/create>: Bad Request ("create ${GOOGLE_CREDENTIALS_PATH}: "${GOOGLE_CREDENTIALS_PATH}" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path")
docker_example_daemon      |
docker_example_daemon      | Stack Trace:
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py", line 332, in _dequeue_run
docker_example_daemon      |     instance.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 161, in launch_run       
docker_example_daemon      |     self._launch_container_with_command(run, docker_image, command)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 113, in _launch_container_with_command
docker_example_daemon      |     **container_context.container_kwargs,
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/models/containers.py", line 917, in create
docker_example_daemon      |     resp = self.client.api.create_container(**create_kwargs)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 431, in create_container
docker_example_daemon      |     return self.create_container_from_config(config, name, platform)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 448, in create_container_from_config   
docker_example_daemon      |     return self._result(res, True)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 274, in _result
docker_example_daemon      |     self._raise_for_status(response)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 270, in _raise_for_status
docker_example_daemon      |     raise create_api_error_from_http_exception(e) from e
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception   
docker_example_daemon      |     raise cls(e, response=response, explanation=explanation) from e
docker_example_daemon      |
docker_example_daemon      | The above exception was caused by the following exception:
docker_example_daemon      | requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.43/containers/create>
docker_example_daemon      |
docker_example_daemon      | Stack Trace:
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 268, in _raise_for_status
docker_example_daemon      |     response.raise_for_status()
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 1021, in raise_for_status
docker_example_daemon      |     raise HTTPError(http_error_msg, response=self)
docker_example_daemon      |
docker_example_daemon      |
Copy code
docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.43/containers/create>: Bad Request ("create ${GOOGLE_CREDENTIALS_PATH}: "${GOOGLE_CREDENTIALS_PATH}" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path")
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py", line 332, in _dequeue_run
    instance.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))
  File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 161, in launch_run
    self._launch_container_with_command(run, docker_image, command)
  File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 113, in _launch_container_with_command
    **container_context.container_kwargs,
  File "/usr/local/lib/python3.7/site-packages/docker/models/containers.py", line 917, in create
    resp = self.client.api.create_container(**create_kwargs)
  File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 431, in create_container
    return self.create_container_from_config(config, name, platform)
  File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 448, in create_container_from_config
    return self._result(res, True)
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/usr/local/lib/python3.7/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
The above exception was caused by the following exception:
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.43/containers/create>
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
i guess my env var is not making it in
a

Aaron T

08/02/2023, 4:44 PM
You can pass a
.env
file into your docker container using docker-compose
env_file
https://docs.docker.com/compose/compose-file/05-services/#env_file
a

Andrew Maguire

08/02/2023, 4:45 PM
Yeah that's what I'm doing elsewhere. I'm just not sure how to make the env bar available in the dagster.yaml file.
a

Aaron T

08/02/2023, 4:46 PM
why does it need to be in your dagster.yaml file? Why not put it in your resources
For some reason the file itself was not getting mounted into one of the containers
But I would rather not have to hardcode that local path
a

Aaron T

08/02/2023, 4:47 PM
Well, if that doesn't work then I would say your local env variable is not what you expect it to be
a

Andrew Maguire

08/02/2023, 4:48 PM
The env variable is as expected, I was logging them to debug. But the file was not getting mounted properly so when it went to look for it file was not found
a

Aaron T

08/02/2023, 4:52 PM
in the same powershell you are running docker compose what does
$env:GOOGLE_CREDENTIALS_PATH
give you?
a

Andrew Maguire

08/02/2023, 4:53 PM
i can see the file mounted in these containers
but its like the job maybe is launching its own container and the file in not available in there
it all works if i mount the path in the run_launcher
Copy code
run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
      - ANOMSTACK_PROJECT_ID
      - ANOMSTACK_DATASET
      - ANOMSTACK_TABLE
      - ANOMSTACK_BUCKET_NAME
      - ANOMSTACK_SLACK_WEBHOOK_URL
      - ANOMSTACK_IGNORE_EXAMPLES
      #- GOOGLE_CREDENTIALS_PATH
    network: docker_example_network
    container_kwargs:
      volumes: # Make docker client accessible to any launched containers as well
        #- C:/Users/andre/Documents/conf/andrewm4894-4d8b5a41b891.json:/gcp_credentials.json # <--works
        #- ${GOOGLE_CREDENTIALS_PATH}:/gcp_credentials.json # <--doesn't work
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/io_manager_storage:/tmp/io_manager_storage
a

Aaron T

08/02/2023, 4:54 PM
ahh I see... then you probably should try adding the variable into the daemon
a

Andrew Maguire

08/02/2023, 4:55 PM
yeah thats what i was thinking but unsure exactly how or where
a

Aaron T

08/02/2023, 4:56 PM
probably the scheduler or run_coordinator
a

Andrew Maguire

08/02/2023, 4:56 PM
thought thats what i was doing there
a

Aaron T

08/02/2023, 4:56 PM
you can also exec -it into the containers to see what is actually there
a

Andrew Maguire

08/03/2023, 12:32 PM
i can see that the file is not available in the containers actually running the job for some reason
i think the root cause is that
${GOOGLE_APPLICATION_CREDENTIALS}
is not getting expanded to be the
GOOGLE_APPLICATION_CREDENTIALS
from my
.env
file.
Copy code
scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
      - ANOMSTACK_PROJECT_ID
      - ANOMSTACK_DATASET
      - ANOMSTACK_TABLE
      - ANOMSTACK_BUCKET_NAME
      - ANOMSTACK_SLACK_WEBHOOK_URL
      - ANOMSTACK_IGNORE_EXAMPLES
      - GOOGLE_APPLICATION_CREDENTIALS
    network: docker_example_network
    container_kwargs:
      volumes: # Make docker client accessible to any launched containers as well
        - ${GOOGLE_APPLICATION_CREDENTIALS}:/gcp_credentials.json # <--doesn't work
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/io_manager_storage:/tmp/io_manager_storage

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      hostname: docker_example_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      hostname: docker_example_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      hostname: docker_example_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432
this is the error i get:
Copy code
docker_example_daemon      | 2023-08-03 12:24:09 +0000 - dagster.daemon.QueuedRunCoordinatorDaemon - ERROR - Caught an unrecoverable error while dequeuing the run. Marking the run as failed and dropping it from the queue: docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.43/containers/create>: Bad Request ("create ${GOOGLE_APPLICATION_CREDENTIALS}: "${GOOGLE_APPLICATION_CREDENTIALS}" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path")
this feels almost like a potential bug or something in dagster (more likely i am misunderstanding something about
dagster.yaml
) but im at point where myself and chatgpt are wondering if we need some crazy custom script approach to expand the env vars in my
dagster.yaml
as part of the Dockerfile or something but that feels very hacky. https://sharegpt.com/c/yB8jVFU
3 Views