Hello I use docker compose locally and I assumed bind mounti dagster #ask-community

Hello, I use docker compose locally and I assumed ...

Hiroto Yamakawa

08/28/2023, 11:51 AM

Hello, I use docker compose locally and I assumed bind mounting the directory where my assets are defined would allow me to apply any code changes directly by clicking on '`reload`' on Dagit, but it doesn't seem to work. the only way is to stop and restart the container

user_code

. Am I missing something in my mental model? Initially, I was copying that directory during the image built and this method works, but I want to avoid having to

docker compose build

and

docker compose up

every time I change something.

Hiroto Yamakawa

08/28/2023, 11:55 AM

I guess this is discussed here https://github.com/dagster-io/dagster/issues/12581

🎉 1

daniel

08/28/2023, 12:06 PM

Hi Hiroto, that’s right - there are instructions at the linked discussion here about how to set this up: https://github.com/dagster-io/dagster/discussions/14709

Hiroto Yamakawa

08/29/2023, 1:11 PM

@daniel Thanks for confirming. I guess it's not really a dagster issue, but any suggestion to get $HOME or $(pwd) work with the

container_kwargs

? in prod I don't have the issue since the project folder will be within the image and the aws role configuration is handled by ECS directly

Copy code

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - [...]
    networks:
      - [...]
    container_kwargs:
      volumes: # Make docker client accessible to any launched containers as well
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/io_manager_storage:/tmp/io_manager_storage
        - $HOME/.aws:/root/.aws
        - $(pwd)/bi_dagster:/opt/dagster/dagster_home/bi_dagster

daniel

08/29/2023, 1:21 PM

I think you would need to do a pre-filtering step that templates in the values for that to work unfortunately

❤️ 1

Hiroto Yamakawa

08/29/2023, 1:21 PM

Noted, I will have a look at it ! thank you 🙏

Hiroto Yamakawa

08/30/2023, 6:58 AM

If that can help anyone. Not the cleanest, since the build depend on a makefile command, but it works. in dagster.yaml

Copy code

container_kwargs:
  volumes: # Make docker client accessible to any launched containers as well
    - /var/run/docker.sock:/var/run/docker.sock
    - /tmp/io_manager_storage:/tmp/io_manager_storage
    - ${HOME_LOCAL}/.aws:/root/.aws  
    - ${PWD_LOCAL}/bi_dagster:/opt/dagster/dagster_home/bi_dagster

In my DockerFile

Copy code

FROM dagster as dagster-dev
COPY ./dagster.dev.yaml dagster.yaml
#REPLACE $HOME_LOCAL and $PWD_LOCAL
COPY .env_build .env
RUN chmod +x .env
RUN . $DAGSTER_HOME/.env && eval "echo \"$(cat dagster.yaml)\"" > dagster.yaml

and in my Makefile,

Copy code

multi-build:
   @echo "HOME_LOCAL=$(HOME)" > .env_build
   @echo "PWD_LOCAL=$(PWD)">> .env_build
   docker compose -f docker-compose.yml build
   @rm .env_build

As an alternative solution: I am pretty sure you can add them as Env variable in your docker compose, and then

Copy code

command: 
eval "echo \"$(cat dagster.yaml)\"" > dagster.yaml && dagster-daemon run

Hiroto Yamakawa

08/30/2023, 7:07 AM

for docker compose :

Copy code

command: >
  sh -c 'eval "echo \"$(cat dagster.yaml)\"" > dagster.yaml &&
          dagster code-server start -h 0.0.0.0 -p 4000 -m bi_dagster'

Open in Slack

Previous Next