is there a way for me to have <https github com andrewm4894 dagster #ask-community

is there a way for me to have <this> be driven by ...

Andrew Maguire

08/02/2023, 4:03 PM

is there a way for me to have this be driven by an env var so its not hardcoded in

dagster.yaml

Andrew Maguire

08/02/2023, 4:03 PM

in docker compose im able to mount it via an env var like this https://github.com/andrewm4894/anomstack/blob/c55a455ea56723c659d9886cad642edc52d10042/docker-compose.yaml#L29

Andrew Maguire

08/02/2023, 4:04 PM

just wondering if some tricks or way to do similar in dagster.yaml so i dont have to hardcode it like above.

Andrew Maguire

08/02/2023, 4:04 PM

idea being once a user sets

GOOGLE_APPLICATION_CREDENTIALS

env var to point at creds file then they are good to go

Andrew Maguire

08/02/2023, 4:08 PM

when i try use env var like this https://github.com/andrewm4894/anomstack/commit/389df8ac8ffb89a9caa52a3d7c538d970c964b18

Andrew Maguire

08/02/2023, 4:08 PM

i see this errror:

Copy code

docker_example_daemon      | 2023-08-02 16:08:05 +0000 - dagster.daemon.QueuedRunCoordinatorDaemon - ERROR - Caught an unrecoverable error while dequeuing the run. Marking the run as failed and dropping it from the queue: docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.43/containers/create>: Bad Request ("create ${GOOGLE_CREDENTIALS_PATH}: "${GOOGLE_CREDENTIALS_PATH}" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path")
docker_example_daemon      |
docker_example_daemon      | Stack Trace:
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py", line 332, in _dequeue_run
docker_example_daemon      |     instance.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 161, in launch_run       
docker_example_daemon      |     self._launch_container_with_command(run, docker_image, command)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 113, in _launch_container_with_command
docker_example_daemon      |     **container_context.container_kwargs,
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/models/containers.py", line 917, in create
docker_example_daemon      |     resp = self.client.api.create_container(**create_kwargs)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 431, in create_container
docker_example_daemon      |     return self.create_container_from_config(config, name, platform)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 448, in create_container_from_config   
docker_example_daemon      |     return self._result(res, True)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 274, in _result
docker_example_daemon      |     self._raise_for_status(response)
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 270, in _raise_for_status
docker_example_daemon      |     raise create_api_error_from_http_exception(e) from e
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception   
docker_example_daemon      |     raise cls(e, response=response, explanation=explanation) from e
docker_example_daemon      |
docker_example_daemon      | The above exception was caused by the following exception:
docker_example_daemon      | requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.43/containers/create>
docker_example_daemon      |
docker_example_daemon      | Stack Trace:
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 268, in _raise_for_status
docker_example_daemon      |     response.raise_for_status()
docker_example_daemon      |   File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 1021, in raise_for_status
docker_example_daemon      |     raise HTTPError(http_error_msg, response=self)
docker_example_daemon      |
docker_example_daemon      |

Andrew Maguire

08/02/2023, 4:09 PM

Copy code

docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.43/containers/create>: Bad Request ("create ${GOOGLE_CREDENTIALS_PATH}: "${GOOGLE_CREDENTIALS_PATH}" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path")
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/run_coordinator/queued_run_coordinator_daemon.py", line 332, in _dequeue_run
    instance.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))
  File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 161, in launch_run
    self._launch_container_with_command(run, docker_image, command)
  File "/usr/local/lib/python3.7/site-packages/dagster_docker/docker_run_launcher.py", line 113, in _launch_container_with_command
    **container_context.container_kwargs,
  File "/usr/local/lib/python3.7/site-packages/docker/models/containers.py", line 917, in create
    resp = self.client.api.create_container(**create_kwargs)
  File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 431, in create_container
    return self.create_container_from_config(config, name, platform)
  File "/usr/local/lib/python3.7/site-packages/docker/api/container.py", line 448, in create_container_from_config
    return self._result(res, True)
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 274, in _result
    self._raise_for_status(response)
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/usr/local/lib/python3.7/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
The above exception was caused by the following exception:
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: <http+docker://localhost/v1.43/containers/create>
  File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

Andrew Maguire

08/02/2023, 4:09 PM

i guess my env var is not making it in

Andrew Maguire

08/02/2023, 4:18 PM

can see here what i am trying to do: https://github.com/andrewm4894/anomstack/commit/c924b2cb20e2b3f2347f7c146a8e4b3408be7116

Aaron T

08/02/2023, 4:44 PM

You can pass a

.env

file into your docker container using docker-compose

env_file

https://docs.docker.com/compose/compose-file/05-services/#env_file

Andrew Maguire

08/02/2023, 4:45 PM

Yeah that's what I'm doing elsewhere. I'm just not sure how to make the env bar available in the dagster.yaml file.

Aaron T

08/02/2023, 4:46 PM

why does it need to be in your dagster.yaml file? Why not put it in your resources

Andrew Maguire

08/02/2023, 4:46 PM

Eg adding here https://github.com/andrewm4894/anomstack/blob/c924b2cb20e2b3f2347f7c146a8e4b3408be7116/docker-compose.yaml#L42

Andrew Maguire

08/02/2023, 4:46 PM

For some reason the file itself was not getting mounted into one of the containers

Andrew Maguire

08/02/2023, 4:47 PM

I have found if I hard code the mount like this, it works https://github.com/andrewm4894/anomstack/blob/c924b2cb20e2b3f2347f7c146a8e4b3408be7116/dagster.yaml#L27

Andrew Maguire

08/02/2023, 4:47 PM

But I would rather not have to hardcode that local path

Aaron T

08/02/2023, 4:47 PM

Well, if that doesn't work then I would say your local env variable is not what you expect it to be

Andrew Maguire

08/02/2023, 4:48 PM

The env variable is as expected, I was logging them to debug. But the file was not getting mounted properly so when it went to look for it file was not found

Aaron T

08/02/2023, 4:52 PM

in the same powershell you are running docker compose what does

$env:GOOGLE_CREDENTIALS_PATH

give you?

Andrew Maguire

08/02/2023, 4:53 PM

i can see the file mounted in these containers

Andrew Maguire

08/02/2023, 4:53 PM

but its like the job maybe is launching its own container and the file in not available in there

Andrew Maguire

08/02/2023, 4:54 PM

it all works if i mount the path in the run_launcher

Copy code

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
      - ANOMSTACK_PROJECT_ID
      - ANOMSTACK_DATASET
      - ANOMSTACK_TABLE
      - ANOMSTACK_BUCKET_NAME
      - ANOMSTACK_SLACK_WEBHOOK_URL
      - ANOMSTACK_IGNORE_EXAMPLES
      #- GOOGLE_CREDENTIALS_PATH
    network: docker_example_network
    container_kwargs:
      volumes: # Make docker client accessible to any launched containers as well
        #- C:/Users/andre/Documents/conf/andrewm4894-4d8b5a41b891.json:/gcp_credentials.json # <--works
        #- ${GOOGLE_CREDENTIALS_PATH}:/gcp_credentials.json # <--doesn't work
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/io_manager_storage:/tmp/io_manager_storage

Aaron T

08/02/2023, 4:54 PM

ahh I see... then you probably should try adding the variable into the daemon

Andrew Maguire

08/02/2023, 4:55 PM

yeah thats what i was thinking but unsure exactly how or where

Andrew Maguire

08/02/2023, 4:56 PM

https://github.com/andrewm4894/anomstack/blob/c924b2cb20e2b3f2347f7c146a8e4b3408be7116/docker-compose.yaml#L90

Aaron T

08/02/2023, 4:56 PM

probably the scheduler or run_coordinator

Andrew Maguire

08/02/2023, 4:56 PM

thought thats what i was doing there

Aaron T

08/02/2023, 4:56 PM

you can also exec -it into the containers to see what is actually there

Andrew Maguire

08/03/2023, 12:32 PM

i can see that the file is not available in the containers actually running the job for some reason

Andrew Maguire

08/03/2023, 12:33 PM

i think the root cause is that

${GOOGLE_APPLICATION_CREDENTIALS}

is not getting expanded to be the

GOOGLE_APPLICATION_CREDENTIALS

from my

.env

file.

Copy code

scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
      - ANOMSTACK_PROJECT_ID
      - ANOMSTACK_DATASET
      - ANOMSTACK_TABLE
      - ANOMSTACK_BUCKET_NAME
      - ANOMSTACK_SLACK_WEBHOOK_URL
      - ANOMSTACK_IGNORE_EXAMPLES
      - GOOGLE_APPLICATION_CREDENTIALS
    network: docker_example_network
    container_kwargs:
      volumes: # Make docker client accessible to any launched containers as well
        - ${GOOGLE_APPLICATION_CREDENTIALS}:/gcp_credentials.json # <--doesn't work
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/io_manager_storage:/tmp/io_manager_storage

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      hostname: docker_example_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      hostname: docker_example_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      hostname: docker_example_postgresql
      username:
        env: DAGSTER_POSTGRES_USER
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      db_name:
        env: DAGSTER_POSTGRES_DB
      port: 5432

Andrew Maguire

08/03/2023, 12:34 PM

this is the error i get:

Copy code

docker_example_daemon      | 2023-08-03 12:24:09 +0000 - dagster.daemon.QueuedRunCoordinatorDaemon - ERROR - Caught an unrecoverable error while dequeuing the run. Marking the run as failed and dropping it from the queue: docker.errors.APIError: 400 Client Error for <http+docker://localhost/v1.43/containers/create>: Bad Request ("create ${GOOGLE_APPLICATION_CREDENTIALS}: "${GOOGLE_APPLICATION_CREDENTIALS}" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path")

Andrew Maguire

08/03/2023, 12:37 PM

this feels almost like a potential bug or something in dagster (more likely i am misunderstanding something about

dagster.yaml

) but im at point where myself and chatgpt are wondering if we need some crazy custom script approach to expand the env vars in my

dagster.yaml

as part of the Dockerfile or something but that feels very hacky. https://sharegpt.com/c/yB8jVFU

Andrew Maguire

08/03/2023, 12:38 PM

am trying to see if there is something i am misunderstanding in here: https://docs.dagster.io/guides/dagster/using-environment-variables-and-secrets#in-yaml-or-config-dictionaries

4 Views

Open in Slack

Previous Next