Umar Hussain
01/19/2023, 8:26 PMdagster-docker
guide and use the dagster api grpc
with a mounted volume to sync changes from my local machine to the host container. It doesn't seem to be working and although the container is running I can't see anything in the logs.
I followed the getting started guide and used
dagster project scaffold --name my-dagster-project
to create a new project.
I also have a dbt
project in the same parent directory with a Docker dir and a docker-compose.yml
are you able to see if I've made a school boy error somewhere? if you need a full MCVE let me know and I'll create dummy repo
├── Docker
│ ├── dagster.DockerFile
│ └── usercode.DockerFile
├── dagster-project
├── dbt-project
└── docker-compose.yml
from my understand any changes to dagster-project
should reflect on dagster running inside my docker image.
-- docker file & compose in this thread.#/Docker/usercode.Docker
FROM python:3.9-slim
# Checkout and install dagster libraries needed to run the gRPC server
# exposing your repository to dagit and dagster-daemon, and to load the DagsterInstance
RUN pip install \
dagster \
dagster-postgres \
dagster-docker
# install system dependencies for pyodbc
RUN apt-get update \
&& apt-get -y install gcc \
&& apt-get -y install g++ \
&& apt-get -y install unixodbc unixodbc-dev \
&& apt-get clean
ENV DAGSTER_HOME=/opt/dagster/dagster-project/
ENV DBT_PROJECT_PATH=/opt/dagster/dbt-project/
ENV DBT_PROFILES=/opt/dagster/dbt-project/
WORKDIR $DAGSTER_HOME
RUN mkdir -p ${DAGSTER_HOME}
RUN mkdir ${DBT_PROJECT_PATH}
COPY ../dagster-project/ /opt/dagster/dagster-project
COPY ../dbt-project /opt/dagster/dbt-project/
RUN pip install -e ".[dev]"
# Run dagster gRPC server on port 4000
EXPOSE 4000
# Using CMD rather than ENTRYPOINT allows the command to be overridden in
# run launchers or executors to run other commands using this image
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000"]
# docker-compose.yaml
version: '3.9'
services:
postgres-db:
image: postgres:latest
container_name: postgres_dagit
volumes:
- ./db:/var/lib/postgresql/data
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
ports:
- "5432:5432"
restart: always
networks:
- retail_dagster_network
docker_retail_code:
build:
context: .
dockerfile: ./Docker/dagster-usercode.DockerFile
container_name: docker_retail_code
image: docker_retail_code_image
restart: always
environment:
DAGSTER_POSTGRES_USER: "postgres"
DAGSTER_POSTGRES_PASSWORD: "postgres"
DAGSTER_POSTGRES_DB: "postgres"
DAGSTER_CURRENT_IMAGE: "docker_retail_code_image"
networks:
- retail_dagster_network
volumes:
- ./pos-pipeline:/opt/dagster/dagster-project
- ./retail_pipeline_dbt:/opt/dagster/dbt-project
docker_retail_dagit:
build:
context: .
dockerfile: ./Docker/dagit.DockerFile
entrypoint:
- dagit
- -h
- "0.0.0.0"
- -p
- "3000"
container_name: retail_dagit
expose:
- "3000"
ports:
- "3000:3000"
environment:
DAGSTER_POSTGRES_USER: "postgres"
DAGSTER_POSTGRES_PASSWORD: "postgres"
DAGSTER_POSTGRES_DB: "postgres"
volumes: # Make docker client accessible so we can terminate containers from dagit
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/io_manager_storage:/tmp/io_manager_storage
networks:
- retail_dagster_network
depends_on:
- postgres-db
- docker_retail_code
docker_retail_daemon:
build:
context: .
dockerfile: ./Docker/dagit.DockerFile
entrypoint:
- dagster-daemon
- run
restart: on-failure
environment:
DAGSTER_POSTGRES_USER: "postgres"
DAGSTER_POSTGRES_PASSWORD: "postgres"
DAGSTER_POSTGRES_DB: "postgres"
volumes: # Make docker client accessible so we can launch containers using host docker
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/io_manager_storage:/tmp/io_manager_storage
networks:
- retail_dagster_network
depends_on:
- postgres-db
- docker_retail_code
volumes:
postgres-db:
driver: local
networks:
retail_dagster_network:
driver: bridge
name: retail_dagster_network
chris
01/21/2023, 1:23 AMUmar Hussain
01/24/2023, 5:19 PMchris
01/25/2023, 12:10 AMjohann
01/25/2023, 1:05 AMUmar Hussain
01/25/2023, 6:35 PM