https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Dante Oz

02/13/2024, 4:29 PM
I’m trying to under the dagster architecture and ops requirements. Based on Open Source deployment architecture page, my understanding is that the Daemon is responsible for launching Runs which execute in ephemeral processes. Does this mean that pipeline code can be updated by replacing the code location container, while asset and jobs are running? For now I’m assuming a single node deployment with separate containers for the webserver, daemon, and code location. Edit: More precisely are in progress runs dependent on the code location container such that if that container is terminated those runs will be killed. Answered: Use the DockerRunLauncher.
r

Riccardo Amadio

02/13/2024, 4:55 PM
Every time that dagster launch a pipeline it takes the code from the code location to a container or pod k8. So if you change the code during execution, the code inside pod or container it'll be not updated.
d

Dante Oz

02/13/2024, 5:15 PM
I’m more so asking about whether the ephemeral process is colocated or depend on the code location container. If the code location container is terminated does that kill the in process run?
I see now in the docs that the DockerRunLauncher should be used. Run are launched in containers which inherit the config of the code location, that way when the code location container is updated new Runs will use the updated code location. The existing Run containers will continue executing uninterrupted by the code location server update.
dagster yay 1
r

Riccardo Amadio

02/14/2024, 8:37 AM
Yes, if you change the code on the repository on GitHub or GitLab or whatever you want, the docker container that it's running the pipeline it'll go on, for consistency reason. It's also the same behavior of every scheduler/orchestrator.
b

Brett

03/28/2024, 2:22 AM
The ask-ai isnt clear about the docker run launcher, in the sense that do we need a separate docker file for each job, or will dagster build a container with the code location at run time and launch the code? If the latter, how does it know which python version to use, and what requirements to install for a job
d

Dante Oz

03/28/2024, 3:57 AM
@Brett Yeah this process could be better documented. The deployment pattern is documented here, and you can find an example docker compose file here. The DockerRunLauncher will use the Docker image specified in the DAGSTER_CURRENT_IMAGE environment variable to launcher the run. This is usually just set to the same Docker image as the Dagster Code Location.
b

Brett

03/28/2024, 1:47 PM
@Dante Oz thanks for sharing! I think I got things setup, but I’m running into this issue now:
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Did you experience anything similar?
d

Dante Oz

03/28/2024, 3:54 PM
I don’t recall running into the error. Where is this error showing up? Is it in the Dagster UI or Docker run logs? If it’s the logs it’d be helpful to know which container is appears under.
b

Brett

03/28/2024, 4:00 PM
Actually fixed that, but running into this now:
Copy code
docker.errors.ImageNotFound: 404 Client Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=dagster-docker-runner>: Not Found ("pull access denied for dagster-docker-runner, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
The image is local on the codeserver, but for some reason it seems like dagster is trying to pull it remotely ?
d

Dante Oz

03/28/2024, 4:19 PM
Does your Docker Compose file have an explicit
image:
field for the code location service?
b

Brett

03/28/2024, 4:24 PM
im using Dockerfile instead actually. I prebuilt an image, and then in
dagster.yaml
I have:
Copy code
run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    image: "dagster-docker-runner:latest"
I even tried adding the variable to the Dockerfile I use for the codeserver:
ENV DAGSTER_CURRENT_IMAGE=dagster-docker-runner
d

Dante Oz

03/28/2024, 4:50 PM
Can you try adding the env var to the daemon container? Also I was using the
QueuedRunCoordinator
.
b

Brett

03/28/2024, 4:52 PM
yeah each container has it’s own copy of dagster.yaml. I’ll probably swap to EFS at some point, but just wanted to get some base level of infra up 🙂. it still doesn’t work either way !
d

Dante Oz

03/28/2024, 5:34 PM
Try removing the
image:
line from the Dagster yaml and only setting
DAGSTER_CURRENT_IMAGE=dagster-docker-runner
. Also did you set a volume bind for
/var/run/docker.sock
.
My configuration is: 1. QueuedRunCoordinator 2.
DAGSTER_CURRENT_IMAGE
environment variable set for my
daemon
and
code-location
containers. 3. A volume bind
/var/run/docker.sock:/var/run/docker.sock
set for the
daemon
container.
If you’re not using the QueuedRunCoordinator then you will probably have to duplicate the volume bind and environment variable configuration in your web server container as both the web server and daemon services will call the run launcher. With the QueuedRunCoordinator all run launches originate from the daemon service.
b

Brett

03/28/2024, 6:11 PM
These are my invocation commands to start up each container service:
Copy code
docker run -v /var/run/docker.sock:/var/run/docker.sock --restart always -d --name dagster-daemon dagster-daemon dagster-daemon run

docker run -v /var/run/docker.sock:/var/run/docker.sock --restart always -d -p 3000:3000 --name dagster-webserver dagster-webserver dagster-webserver -h 0.0.0.0 -p 3000

docker run -v /var/run/docker.sock:/var/run/docker.sock --restart always -d -p 4000:4000 --name dagster-codeserver dagster-codeserver dagster code-server start -f all_code_entry.py --host 0.0.0.0 --port 4000
The
dagster-docker-runner
image definitely exists on the code server:
Copy code
$ docker images
REPOSITORY              TAG         IMAGE ID       CREATED          SIZE
dagster-docker-runner   latest      e11178232ace   3 seconds ago    360MB
dagster-codeserver      latest      dcdff35e3050   5 minutes ago    361MB
<none>                  <none>      8272e767b9b0   17 minutes ago   361MB
<none>                  <none>      37a4c1a17974   2 hours ago      361MB
<none>                  <none>      6236027e0d24   2 hours ago      361MB
python                  3.12-slim   23cd4350f4bd   6 days ago       130MB
I tried moving
DAGSTER_CURRENT_IMAGE
around to dagster.yaml and each of the service containers (1 x daemon, codeserver, UI webserver). Still getting that login warning. I think dagster is trying to pull a container from the registry instead of locally. That is my guess. Are your containers in dockerhub?
d

Dante Oz

03/28/2024, 7:22 PM
No my containers are local. Try adding
-e DAGSTER_CURRENT_IMAGE='dagster-codeserver'
to the daemon and code-location invocations.
b

Brett

03/28/2024, 8:09 PM
Same issue !
Copy code
docker.errors.ImageNotFound: 404 Client Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=dagster-codeserver>: Not Found ("pull access denied for dagster-codeserver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
3 Views