chrispc
03/04/2022, 1:54 PM"File "/usr/local/lib/python3.9/site-packages/dagster_docker/docker_run_launcher.py", line 83, in _get_docker_image
raise Exception(“No docker image specified by the instance config or repository”)” , my docker-compose :
version: "3.7"
services:
dagster_pipelines:
build:
context: .
dockerfile: ./Dockerfile_pipelines
container_name: dagster_pipelines
image: dagster_pipeline_image
restart: always
ports:
- "4000:4000"
env_file:
- .env
volumes:
- ./-----.json:/var/-----.json
networks:
- dagster_network
dagster_dagit:
build:
context: .
dockerfile: ./Dockerfile_dagster
entrypoint:
- dagit
- -h
- "0.0.0.0"
- -p
- "3000"
- -w
- workspace.yaml
container_name: dagster_dagit
image: dagster_app
expose:
- "3000"
ports:
- "3000:3000"
env_file:
- .env
volumes: # Make docker client accessible so we can terminate containers from dagit
- /var/run/docker.sock:/var/run/docker.sock
networks:
- dagster_network
depends_on:
- dagster_pipelines
dagster_daemon:
entrypoint:
- dagster-daemon
- run
container_name: dagster_daemon
image: dagster_app
restart: on-failure
env_file:
- .env
volumes: # Make docker client accessible so we can launch containers using host docker
- /var/run/docker.sock:/var/run/docker.sock
networks:
- dagster_network
depends_on:
- dagster_pipelines
networks:
dagster_network:
driver: bridge
name: dagster_network
Dockerfile_pipelines:
FROM python:3.9.10-slim
RUN pip install -U pip setuptools
RUN pip install --no-cache-dir\
pandas==1.3.3 \
dagster==0.14.2 \
dagster-graphql==0.14.2 \
dagit==0.14.2 \
dagster-postgres==0.14.2 \
dagster-docker==0.14.2 \
dagster-gcp==0.14.2 \
gcsfs==2022.2.0 \
dateparser==1.0.0 \
db-dtypes \
great-expectations==0.14.4 \
python-decouple==3.4 \
avro
WORKDIR /var
COPY -------.json ./
ENV GOOGLE_APPLICATION_CREDENTIALS=/var/-----.json
WORKDIR /opt/dagster/app/
# Add repository code
RUN mkdir CAPS
RUN mkdir great_expectations_root
RUN mkdir -p local_artifact_storage/storage/
COPY CAPS/ CAPS/
COPY great_expectations_root/ great_expectations_root/
COPY config/local_docker/dagster.yaml config/local_docker/workspace.yaml setup.py ./
ENV DAGSTER_HOME=/opt/dagster/app/
RUN pip install -e .
# Run dagster gRPC server on port 4000
EXPOSE 4000
# Using CMD rather than ENTRYPOINT allows the command to be overridden in
# run launchers or executors to run other commands using this image
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "CAPS/caps.py"]
Dockerfile_dagster:
FROM python:3.9.10-slim
ENV DAGSTER_HOME=/opt/dagster/dagster_home/
ENV DAGSTER_PG_USERNAME=-----
ENV DAGSTER_PG_PASSWORD=-----
ENV DAGSTER_PG_HOST=----
ENV DAGSTER_PG_DB=---
RUN mkdir -p $DAGSTER_HOME
RUN pip install -U pip setuptools
RUN pip install --no-cache-dir\
dagster==0.14.2 \
dagster-graphql==0.14.2 \
dagit==0.14.2 \
dagster-postgres==0.14.2 \
dagster-docker==0.14.2 \
dagster-gcp==0.14.2
# Set $DAGSTER_HOME and copy dagster instance and workspace YAML there
COPY config/local_docker/dagster.yaml config/local_docker/workspace.yaml $DAGSTER_HOME
WORKDIR $DAGSTER_HOME
EXPOSE 3000
ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "3000"]
worskpace.yaml
load_from:
- grpc_server:
host: dagster_pipelines
port: 4000
location_name: "dagster_pipelines"
dagster.yaml
scheduler:
module: dagster.core.scheduler
class: DagsterDaemonScheduler
run_launcher:
module: dagster_docker
class: DockerRunLauncher
config:
env_vars:
- DAGSTER_PG_USERNAME
- DAGSTER_PG_PASSWORD
- DAGSTER_PG_HOST
- DAGSTER_PG_DB
network: dagster_network
container_kwargs:
auto_remove: true
run_coordinator:
module: dagster.core.run_coordinator
class: QueuedRunCoordinator
config:
max_concurrent_runs: 25
run_storage:
module: dagster_postgres.run_storage
class: PostgresRunStorage
config:
postgres_db:
username:
env: DAGSTER_PG_USERNAME
password:
env: DAGSTER_PG_PASSWORD
hostname:
env: DAGSTER_PG_HOST
db_name:
env: DAGSTER_PG_DB
port: 5432
Ant idea what is happening? Thank you in advance.daniel
03/04/2022, 4:19 PMchrispc
03/04/2022, 6:32 PMGeorge Pearse
03/25/2022, 1:12 PMdocker-compose pull
would it be
image_name: <remote-name>/<project-name>/<image-name>
Or just
image_name: <image-name>
?