https://dagster.io/ logo
#ask-community
Title
# ask-community
u

Umar Hussain

01/19/2023, 8:26 PM
hiya I'm attempting to the follow the
dagster-docker
guide and use the
dagster api grpc
with a mounted volume to sync changes from my local machine to the host container. It doesn't seem to be working and although the container is running I can't see anything in the logs. I followed the getting started guide and used
Copy code
dagster project scaffold --name my-dagster-project
to create a new project. I also have a
dbt
project in the same parent directory with a Docker dir and a docker-compose.yml are you able to see if I've made a school boy error somewhere? if you need a full MCVE let me know and I'll create dummy repo
Copy code
├── Docker
│   ├── dagster.DockerFile
│   └── usercode.DockerFile
├── dagster-project
├── dbt-project
└── docker-compose.yml
from my understand any changes to
dagster-project
should reflect on dagster running inside my docker image. -- docker file & compose in this thread.
Copy code
#/Docker/usercode.Docker
FROM python:3.9-slim

# Checkout and install dagster libraries needed to run the gRPC server
# exposing your repository to dagit and dagster-daemon, and to load the DagsterInstance

RUN pip install \
    dagster \
    dagster-postgres \
    dagster-docker

# install system dependencies for pyodbc
RUN apt-get update \
  && apt-get -y install gcc \
  && apt-get -y install g++ \
  && apt-get -y install unixodbc unixodbc-dev \
  && apt-get clean


ENV DAGSTER_HOME=/opt/dagster/dagster-project/
ENV DBT_PROJECT_PATH=/opt/dagster/dbt-project/
ENV DBT_PROFILES=/opt/dagster/dbt-project/

WORKDIR $DAGSTER_HOME


RUN mkdir -p ${DAGSTER_HOME}
RUN mkdir ${DBT_PROJECT_PATH}

COPY ../dagster-project/ /opt/dagster/dagster-project
COPY ../dbt-project /opt/dagster/dbt-project/

RUN pip install -e ".[dev]"

# Run dagster gRPC server on port 4000

EXPOSE 4000

# Using CMD rather than ENTRYPOINT allows the command to be overridden in
# run launchers or executors to run other commands using this image
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000"]
Copy code
# docker-compose.yaml

version: '3.9'

services:
  postgres-db:
    image: postgres:latest
    container_name: postgres_dagit 
    volumes:
      - ./db:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=postgres
    ports:
      - "5432:5432"
    restart: always
    networks:
      - retail_dagster_network

  docker_retail_code:
    build:
      context: .
      dockerfile: ./Docker/dagster-usercode.DockerFile
    container_name: docker_retail_code
    image: docker_retail_code_image
    restart: always
    environment:
      DAGSTER_POSTGRES_USER: "postgres"
      DAGSTER_POSTGRES_PASSWORD: "postgres"
      DAGSTER_POSTGRES_DB: "postgres"
      DAGSTER_CURRENT_IMAGE: "docker_retail_code_image"
    networks:
      - retail_dagster_network
    volumes:
      - ./pos-pipeline:/opt/dagster/dagster-project
      - ./retail_pipeline_dbt:/opt/dagster/dbt-project

  docker_retail_dagit:
    build:
      context: .
      dockerfile: ./Docker/dagit.DockerFile
    entrypoint:
      - dagit
      - -h
      - "0.0.0.0"
      - -p
      - "3000"
    container_name: retail_dagit
    expose:
      - "3000"
    ports:
      - "3000:3000"
    environment:
      DAGSTER_POSTGRES_USER: "postgres"
      DAGSTER_POSTGRES_PASSWORD: "postgres"
      DAGSTER_POSTGRES_DB: "postgres"
    volumes: # Make docker client accessible so we can terminate containers from dagit
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/io_manager_storage:/tmp/io_manager_storage
    networks:
      - retail_dagster_network
    depends_on:
      - postgres-db
      - docker_retail_code

  docker_retail_daemon:
    build:
      context: .
      dockerfile: ./Docker/dagit.DockerFile
    entrypoint:
      - dagster-daemon
      - run
    restart: on-failure
    environment:
      DAGSTER_POSTGRES_USER: "postgres"
      DAGSTER_POSTGRES_PASSWORD: "postgres"
      DAGSTER_POSTGRES_DB: "postgres"
    volumes: # Make docker client accessible so we can launch containers using host docker
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/io_manager_storage:/tmp/io_manager_storage
    networks:
      - retail_dagster_network
    depends_on:
      - postgres-db
      - docker_retail_code
      
      
volumes:
  postgres-db:
    driver: local

networks:
  retail_dagster_network:
    driver: bridge
    name: retail_dagster_network
c

chris

01/21/2023, 1:23 AM
Are you getting a particular error or are you just not seeing your changes show up when they are made?
u

Umar Hussain

01/24/2023, 5:19 PM
sorry @chris totally missed this! ( didn't get a notif!) No error the project runs (but probably not correctly) but any changes don't sync from my local FS to the container
so to sync any changes i need to do a full --build
for prod releases this is fine but for dev work it's painfully slow
c

chris

01/25/2023, 12:10 AM
hey - reaching out to members of the team who have more experience with docker-based deployment
j

johann

01/25/2023, 1:05 AM
If you restart the container does it pick up the changes?
Is docker a requirement for dev? If you’re able to run locally, that often is the lowest friction dev experience
u

Umar Hussain

01/25/2023, 6:35 PM
got it, I'll stick with a non-dockerized exp for dev thanks @johann
4 Views