Hello all, I'm trying to spin up a PoC for my comp...
# ask-community
j
Hello all, I'm trying to spin up a PoC for my company. I started my project from the example project deploy_docker and adapted only two files: repo.py and Dockerfile_user_code. When I try to spin up the docker compose stack I got an grpc error which is saying that the user code server cannot be reached (stack trace in thread). In my repo, I try to make a connection to a sql server database with sqlalchemy and Pyodbc. I have therefore my dockerfile_user_code adapted to the following (see thread). Can someone please help me to solve this issue? Thank you very much for your help.
dagster bot responded by community 1
Stack trace: /usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py586 UserWarning: Error loading repository location example_user_codedagster. core.errors.DagsterUserCodeUnreachableError Could not reach user code server. gRPC Error code: UNAVAILABLE docker_example_dagit | docker_example_dagit | Stack Trace: docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py", line 581, in _load_location docker_example_dagit | location = self._create_location_from_origin(origin) docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py", line 501, in _create_location_from_origin docker_example_dagit | return origin.create_location() docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/origin.py", line 331, in create_location docker_example_dagit | return GrpcServerRepositoryLocation(self) docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/repository_location.py", line 575, in init docker_example_dagit | list_repositories_response = sync_list_repositories_grpc(self.client) docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_api/list_repositories.py", line 19, in sync_list_repositories_grpc docker_example_dagit | api_client.list_repositories(), docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 230, in list_repositories docker_example_dagit | res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest) docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 157, in _query docker_example_dagit | e, timeout=timeout, custom_timeout_message=custom_timeout_message docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 141, in _raise_grpc_exception docker_example_dagit | ) from e docker_example_dagit | docker_example_dagit | The above exception was caused by the following exception: docker_example_dagit | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: docker_example_dagit | status = StatusCode.UNAVAILABLE docker_example_dagit | details = "failed to connect to all addresses" docker_example_dagit | debug_error_string = "{"created":"@1674117761.666041800","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1674117761.666041200","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}" docker_example_dagit | > docker_example_dagit | docker_example_dagit | Stack Trace: docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 154, in _query docker_example_dagit | return self._get_response(method, request=request_type(**kwargs), timeout=timeout) docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 129, in _get_response docker_example_dagit | return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout) docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in call docker_example_dagit | return _end_unary_response_blocking(state, call, False, None) docker_example_dagit | File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking docker_example_dagit | raise _InactiveRpcError(state) docker_example_dagit | docker_example_dagit | location_name=location_name, error_string=error.to_string()
FROM python:3.7-slim
ENV ACCEPT_EULA=Y RUN apt-get update -y && apt-get update \ && apt-get install -y --no-install-recommends curl gcc g++ gnupg unixodbc-dev unixodbc # Add SQL Server ODBC Driver 17 for Ubuntu 18.04 RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \ && curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list \ && apt-get update \ && apt-get install -y --no-install-recommends --allow-unauthenticated msodbcsql17 mssql-tools \ && echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile \ && echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc COPY requirements.txt requirements.txt RUN pip3 install -r requirements.txt # Set $DAGSTER_HOME and copy dagster instance there ENV DAGSTER_HOME=/opt/dagster/dagster_home RUN mkdir -p $DAGSTER_HOME COPY dagster.yaml $DAGSTER_HOME # Add repository code WORKDIR /opt/dagster/app COPY repo.py /opt/dagster/app # Run dagster gRPC server on port 4000 EXPOSE 4000 # CMD allows this to be overridden from run launchers or executors that want # to run other commands against your repository CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"]
d
Hey! Are you running Dagit,the Dagster daemon, and the user code on the same Docker network?
I am using this docker-compose file to run Dagster locally (including a postgres database):
version: '3.7'
services:
# This service runs the postgres DB used by dagster for run storage, schedule storage,
# and event log storage.
docker_example_postgresql:
image: postgres:11
container_name: docker_example_postgresql
environment:
POSTGRES_USER: "postgres_user"
POSTGRES_PASSWORD: "postgres_password"
POSTGRES_DB: "postgres_db"
networks:
- docker_example_network
# This service runs the gRPC server that loads your user code, in both dagit
# and dagster-daemon. By setting DAGSTER_CURRENT_IMAGE to its own image, we tell the
# run launcher to use this same image when launching runs in a new container as well.
# Multiple containers like this can be deployed separately - each just needs to run on
# its own port, and have its own entry in the workspace.yaml file that's loaded by dagit.
docker_example_user_code:
build:
context: .
dockerfile: ./Dockerfile_user_code-dev
container_name: docker_example_user_code
image: docker_example_user_code_image
restart: always
environment:
DAGSTER_POSTGRES_USER: "postgres_user"
DAGSTER_POSTGRES_PASSWORD: "postgres_password"
DAGSTER_POSTGRES_DB: "postgres_db"
DAGSTER_CURRENT_IMAGE: "docker_example_user_code_image"
networks:
- docker_example_network
# This service runs dagit, which loads your user code from the user code container.
# Since our instance uses the QueuedRunCoordinator, any runs submitted from dagit will be put on
# a queue and later dequeued and launched by dagster-daemon.
docker_example_dagit:
build:
context: .
dockerfile: ./Dockerfile_dagster-dev
entrypoint:
- dagit
- -h
- "0.0.0.0"
- -p
- "3000"
- -w
- workspace.yaml
container_name: docker_example_dagit
expose:
- "3000"
ports:
- "3000:3000"
environment:
DAGSTER_POSTGRES_USER: "postgres_user"
DAGSTER_POSTGRES_PASSWORD: "postgres_password"
DAGSTER_POSTGRES_DB: "postgres_db"
volumes: # Make docker client accessible so we can terminate containers from dagit
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/io_manager_storage:/tmp/io_manager_storage
networks:
- docker_example_network
depends_on:
- docker_example_postgresql
- docker_example_user_code
# This service runs the dagster-daemon process, which is responsible for taking runs
# off of the queue and launching them, as well as creating runs from schedules or sensors.
docker_example_daemon:
build:
context: .
dockerfile: ./Dockerfile_dagster-dev
entrypoint:
- dagster-daemon
- run
container_name: docker_example_daemon
restart: on-failure
environment:
DAGSTER_POSTGRES_USER: "postgres_user"
DAGSTER_POSTGRES_PASSWORD: "postgres_password"
DAGSTER_POSTGRES_DB: "postgres_db"
volumes: # Make docker client accessible so we can launch containers using host docker
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/io_manager_storage:/tmp/io_manager_storage
networks:
- docker_example_network
depends_on:
- docker_example_postgresql
- docker_example_user_code
networks:
docker_example_network:
driver: bridge
name: docker_example_network
j
Hey Daniel, thank you for your reply! I have put the two docker compose files (yours and mine) besides each other and they are exactly the same. I haven't touched this file and without any changes (so right from the example project), I could perfectly run the example. So I supposed that when I changed repo.py and Dockerfile_user_code it would work...
d
Hmm that's strange, your Dockerfile is also similar to mine (I don't copy in the dagster.yaml file in the user code image but that shouldn't break anything). What about the user deployment container? what does
docker ps
return? Is it running?
j
Yes, the docker user container is running 🙂
d
I see, I assume you also checked the container's logs? 🙂
j
At the moment, I receive another error (after trying different things). Could not load location repo.py to check for sensors due to the following error: FileNotFoundError: [Errno 2] No such file or directory: '/opt/dagster/dagster_home/repo.py'
This is the original error: /usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py586 UserWarning: Error loading repository location repo.pyFileNotFoundError [Errno 2] No such file or directory: '/opt/dagster/dagster_home/repo.py'
The last error was my fault. A space missing in workspace.yaml... Apologies for the inconvenience
d
No worries! At least you got it fixed now 🙂
j
The last error yes 😄, but unfortunately still struggling with grpc server error
d
Ahh well from what I can see, you are copying your repo.py file into /opt/dagster/app but it is trying to get loaded from /opt/dagster/dagster_home
I am referring to this part of your Dockerfile:
WORKDIR /opt/dagster/app
COPY repo.py /opt/dagster/app
# Run dagster gRPC server on port 4000
EXPOSE 4000
# CMD allows this to be overridden from run launchers or executors that want
# to run other commands against your repository
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"]
I think it's because you are manually setting $DAGSTER_HOME, as I said I don't do this in my Dockerfile (for the user deployment code). Mine looks like this:
FROM python:3.7-slim
# Checkout and install dagster libraries needed to run the gRPC server
# exposing your repository to dagit and dagster-daemon, and to load the DagsterInstance
COPY requirements-user-code.txt requirements-user-code.txt
RUN pip install -r requirements-user-code.txt
WORKDIR /opt/dagster/app
ENV PLATFORM_ENVIRONMENT=DEV
COPY dagster_src /opt/dagster/app/dagster_src
# Run dagster gRPC server on port 4000
EXPOSE 4000
# CMD allows this to be overridden from run launchers or executors that want
# to run other commands against your repository
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "dagster_src/repo.py"]