https://dagster.io/ logo
#dagster-support
Title
# dagster-support
a

Alec Ryan

05/05/2022, 12:49 PM
Hey All, I'm having a difficult time connecting my docker_user_code container to my dagit & daemon container. Seeing the below error. Any thoughts on what the issue could be? I followed this guide: https://docs.dagster.io/deployment/guides/docker
compose
Copy code
version: "3.7"

services:

  docker_user_code:
    build:
      context: .
      dockerfile: ./Dockerfile_user_code
    container_name: docker_user_code
    image: docker_user_code_image
    restart: always
    environment:
      DAGSTER_POSTGRES_USER:
      DAGSTER_POSTGRES_PASSWORD: 
      DAGSTER_POSTGRES_DB: 
      DAGSTER_CURRENT_IMAGE: docker_user_code_image
    networks:
      - docker_network

  dagit:
    build:
      context: . 
      dockerfile: ./Dockerfile_dagster
    entrypoint:
      - dagit
      - -h 
      - "0.0.0.0"
      - -p 
      - "3000"
      - -w 
      - workspace.yaml
    container_name: docker_dagit
    expose:
      - "3000"
    ports:
      - "3000:3000"
    environment:
      DAGSTER_POSTGRES_USER: 
      DAGSTER_POSTGRES_PASSWORD: 
      DAGSTER_POSTGRES_DB: 
    volumes: # Make docker client accessible so we can terminate containers from dagit
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - docker_network
    depends_on:
      - docker_user_code

  docker_daemon:
    build:
      context: .
      dockerfile: ./Dockerfile_dagster
    entrypoint:
      - dagster-daemon
      - run
    container_name: docker_daemon
    restart: on-failure
    environment:
      DAGSTER_POSTGRES_USER:
      DAGSTER_POSTGRES_PASSWORD:
      DAGSTER_POSTGRES_DB: 
    volumes: # Make docker client accessible so we can launch containers using host docker
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - docker_network
    depends_on:
      - docker_user_code

networks:
  docker_network:
    driver: bridge
    name: docker_network
Dockerfile_user_code
Copy code
FROM python:3.7-slim

RUN mkdir -p /opt/dagster/dagster_home /opt/dagster/app
RUN mkdir /opt/dagster/app/nhl_dagster
RUN mkdir /opt/dagster/app/nhl_dbt

COPY requirements.txt /opt/dagster/app
COPY nhl_dbt /opt/dagster/app/nhl_dbt/ 
COPY nhl_dagster /opt/dagster/app/nhl_dagster/
COPY workspace.yaml /opt/dagster/app/

ENV DAGSTER_HOME=/opt/dagster/dagster_home/

COPY dagster.yaml /opt/dagster/dagster_home/

WORKDIR /opt/dagster/app

RUN pip install --upgrade pip
RUN pip install -r requirements.txt

EXPOSE 4000

# CMD allows this to be overridden from run launchers or executors that want
# to run other commands against your repository
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "nhl_dagster/repo.py"]
Dockerfile_dagster:
Copy code
# Dagster libraries to run both dagit and the dagster-daemon. Does not
# need to have access to any pipeline code.

FROM python:3.7-slim

RUN pip install \
    dagster \
    dagster-graphql \
    dagit \
    dagster-postgres \
    dagster-docker

# Set $DAGSTER_HOME and copy dagster instance and workspace YAML there
ENV DAGSTER_HOME=/opt/dagster/dagster_home/

RUN mkdir -p $DAGSTER_HOME

COPY dagster.yaml workspace.yaml $DAGSTER_HOME

WORKDIR $DAGSTER_HOME
j

johann

05/05/2022, 1:53 PM
Could you share your
workspace.yaml
?
a

Alec Ryan

05/05/2022, 1:55 PM
@johann workspace.yaml:
Copy code
load_from:
  # Each entry here corresponds to a service in the docker-compose file that exposes user code.
  - grpc_server:
      host: docker_user_code
      port: 4000
      location_name: "docker_user_code"
j

johann

05/05/2022, 1:59 PM
Nothing is jumping out… if you use the exact docker example does it work on your machine? https://github.com/dagster-io/dagster/blob/0.14.13/examples/deploy_docker/docker-compose.yml
a

Alec Ryan

05/05/2022, 7:25 PM
@johann yes it does
seemingly no issue
In my project, I am using an RDS instance
I assume I don't need to specify a container for that right?
j

johann

05/05/2022, 8:39 PM
You don’t. In your example, are there any logs from the docker_user_code container?
It may be easiest to start from that working example and make one change at a time- e.g. either just change the DB and get that working, or just change the example code image
a

Alec Ryan

05/05/2022, 10:27 PM
Yeah, good call. I'll try to step through that tonight
Thanks for the help
Latest error I've seen @johann
Copy code
Operation name: AssetGraphLiveQuery

Message: 'b900fd88-92d4-4c1d-b96b-d2a870837d70'

Path: ["repositoriesOrError","nodes",0,"latestRunByStep"]

Locations: [{"line":32,"column":3}]
I noticed that repo.py in the example is in the parent directory
my repo is located @ ./nhl_dagster/repo.py
Now I seem to have my pipeline working as expected. I see this error now but I don't think it actually impacts my pipeline:
j

johann

05/06/2022, 3:16 PM
We wouldn’t expect that to impact anything, it’s a clean up after a disconnect
a

Alec Ryan

05/06/2022, 3:18 PM
Okay cool, so this error can just be ignored?
j

johann

05/06/2022, 3:20 PM
Yes
❤️ 1
a

Alec Ryan

05/06/2022, 3:26 PM
thanks for all of the help!
j

Jing Zhang

05/10/2022, 8:40 PM
It's great to know that it doesn't impact the pipelines. @johann Could you share what kind of disconnect generates these error messages? I see them quite often and want to avoid these messages flooding my server log monitors. Thank you.
a

alex

05/10/2022, 8:53 PM
I believe its just normal disconnects, anytime someone using the webapp changes away from a run page for example. Something we need to fix https://github.com/dagster-io/dagster/issues/7832
j

Jing Zhang

05/10/2022, 10:11 PM
Cool thanks @alex
2 Views