https://dagster.io/ logo
#ask-community
Title
# ask-community
c

chrispc

03/04/2022, 1:54 PM
Hi Team (Hi, again @daniel), Could I get some help? I am trying to run dagster as multicontainer mode in my local machine. It was working with V0.12, but I migrated to V0.14 to test the new features, however, now It is not working. I am getting this error:
"File "/usr/local/lib/python3.9/site-packages/dagster_docker/docker_run_launcher.py", line 83, in _get_docker_image
raise Exception(“No docker image specified by the instance config or repository”)” , my docker-compose :
Copy code
version: "3.7"
services:
  dagster_pipelines:
    build:
      context: .
      dockerfile: ./Dockerfile_pipelines
    container_name: dagster_pipelines
    image: dagster_pipeline_image
    restart: always
    ports:
      - "4000:4000"
    env_file:
      - .env
    volumes:
      - ./-----.json:/var/-----.json
    networks:
      - dagster_network

  dagster_dagit:
    build:
      context: .
      dockerfile: ./Dockerfile_dagster
    entrypoint:
      - dagit
      - -h
      - "0.0.0.0"
      - -p
      - "3000"
      - -w
      - workspace.yaml
    container_name: dagster_dagit
    image: dagster_app
    expose:
      - "3000"
    ports:
      - "3000:3000"
    env_file:
      - .env
    volumes: # Make docker client accessible so we can terminate containers from dagit
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - dagster_network
    depends_on:
      - dagster_pipelines
  dagster_daemon:
    entrypoint:
      - dagster-daemon
      - run
    container_name: dagster_daemon
    image: dagster_app
    restart: on-failure
    env_file:
      - .env
    volumes: # Make docker client accessible so we can launch containers using host docker
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - dagster_network
    depends_on:
      - dagster_pipelines
networks:
  dagster_network:
    driver: bridge
    name: dagster_network
Dockerfile_pipelines:
Copy code
FROM python:3.9.10-slim
RUN pip install -U pip setuptools
RUN pip install --no-cache-dir\
    pandas==1.3.3 \
    dagster==0.14.2 \
    dagster-graphql==0.14.2 \
    dagit==0.14.2 \
    dagster-postgres==0.14.2 \
    dagster-docker==0.14.2 \
    dagster-gcp==0.14.2 \
    gcsfs==2022.2.0 \
    dateparser==1.0.0 \
    db-dtypes \
    great-expectations==0.14.4 \
    python-decouple==3.4 \
    avro


WORKDIR /var
COPY -------.json ./
ENV GOOGLE_APPLICATION_CREDENTIALS=/var/-----.json

WORKDIR /opt/dagster/app/
# Add repository code
RUN mkdir CAPS
RUN mkdir great_expectations_root
RUN mkdir -p local_artifact_storage/storage/

COPY CAPS/ CAPS/
COPY great_expectations_root/ great_expectations_root/
COPY config/local_docker/dagster.yaml config/local_docker/workspace.yaml setup.py ./

ENV DAGSTER_HOME=/opt/dagster/app/

RUN pip install -e .

# Run dagster gRPC server on port 4000

EXPOSE 4000

# Using CMD rather than ENTRYPOINT allows the command to be overridden in
# run launchers or executors to run other commands using this image
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "CAPS/caps.py"]
Dockerfile_dagster:
Copy code
FROM python:3.9.10-slim

ENV DAGSTER_HOME=/opt/dagster/dagster_home/
ENV DAGSTER_PG_USERNAME=-----
ENV DAGSTER_PG_PASSWORD=-----
ENV DAGSTER_PG_HOST=----
ENV DAGSTER_PG_DB=---

RUN mkdir -p $DAGSTER_HOME

RUN pip install -U pip setuptools
RUN pip install --no-cache-dir\
    dagster==0.14.2 \
    dagster-graphql==0.14.2  \
    dagit==0.14.2  \
    dagster-postgres==0.14.2  \
    dagster-docker==0.14.2  \
    dagster-gcp==0.14.2

# Set $DAGSTER_HOME and copy dagster instance and workspace YAML there

COPY config/local_docker/dagster.yaml config/local_docker/workspace.yaml $DAGSTER_HOME

WORKDIR $DAGSTER_HOME

EXPOSE 3000

ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "3000"]
worskpace.yaml
Copy code
load_from:
  - grpc_server:
      host: dagster_pipelines
      port: 4000
      location_name: "dagster_pipelines"
dagster.yaml
Copy code
scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_PG_USERNAME
      - DAGSTER_PG_PASSWORD
      - DAGSTER_PG_HOST
      - DAGSTER_PG_DB
    network: dagster_network
    container_kwargs:
      auto_remove: true

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 25

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_PG_USERNAME
      password:
        env: DAGSTER_PG_PASSWORD
      hostname:
        env: DAGSTER_PG_HOST
      db_name:
        env: DAGSTER_PG_DB
      port: 5432
Ant idea what is happening? Thank you in advance.
d

daniel

03/04/2022, 4:19 PM
hi Christian - you'll want to make sure that your user code image is exposing the name of the image to use in the DAGSTER_CURRENT_IMAGE environment variable: https://docs.dagster.io/deployment/guides/docker#launching-runs-in-containers Like in our example here: https://github.com/dagster-io/dagster/blob/0.14.3/examples/deploy_docker/docker-compose.yml#L16-L32
❤️ 1
c

chrispc

03/04/2022, 6:32 PM
Ohh I missed that one. Let me check @daniel, thank you so much
g

George Pearse

03/25/2022, 1:12 PM
How does this work when the image name has a version tag as well that's made available as an environment variable?
Okay string interpolation was simple enough. How does this work with a remote image that can be used in
Copy code
docker-compose pull
would it be
Copy code
image_name: <remote-name>/<project-name>/<image-name>
Or just
Copy code
image_name: <image-name>
?
Ooh think this thread has solved my problem of the last 5 hours. Thanks a lot both.
condagster 1
3 Views