I’m trying to under the dagster architecture and ops requirements. Based on Open Source deployment architecture page, my understanding is that the Daemon is responsible for launching Runs which execute in ephemeral processes. Does this mean that pipeline code can be updated by replacing the code location container, while asset and jobs are running? For now I’m assuming a single node deployment with separate containers for the webserver, daemon, and code location. Edit: More precisely are in progress runs dependent on the code location container such that if that container is terminated those runs will be killed. Answered: Use the DockerRunLauncher.

Every time that dagster launch a pipeline it takes the code from the code location to a container or pod k8. So if you change the code during execution, the code inside pod or container it'll be not updated.

I’m more so asking about whether the ephemeral process is colocated or depend on the code location container. If the code location container is terminated does that kill the in process run?

Yes, if you change the code on the repository on GitHub or GitLab or whatever you want, the docker container that it's running the pipeline it'll go on, for consistency reason. It's also the same behavior of every scheduler/orchestrator.

The ask-ai isnt clear about the docker run launcher, in the sense that do we need a separate docker file for each job, or will dagster build a container with the code location at run time and launch the code? If the latter, how does it know which python version to use, and what requirements to install for a job

@Brett Yeah this process could be better documented. The deployment pattern is documented here, and you can find an example docker compose file here. The DockerRunLauncher will use the Docker image specified in the DAGSTER_CURRENT_IMAGE environment variable to launcher the run. This is usually just set to the same Docker image as the Dagster Code Location.

@Dante Oz thanks for sharing! I think I got things setup, but I’m running into this issue now:

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Did you experience anything similar?

I don’t recall running into the error. Where is this error showing up? Is it in the Dagster UI or Docker run logs? If it’s the logs it’d be helpful to know which container is appears under.

Actually fixed that, but running into this now:

docker.errors.ImageNotFound: 404 Client Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=dagster-docker-runner>: Not Found ("pull access denied for dagster-docker-runner, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The image is local on the codeserver, but for some reason it seems like dagster is trying to pull it remotely ?

Does your Docker Compose file have an explicit

image:

field for the code location service?

im using Dockerfile instead actually. I prebuilt an image, and then in

dagster.yaml

I have:

run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    image: "dagster-docker-runner:latest"

I even tried adding the variable to the Dockerfile I use for the codeserver:

ENV DAGSTER_CURRENT_IMAGE=dagster-docker-runner

Can you try adding the env var to the daemon container? Also I was using the

QueuedRunCoordinator

.

yeah each container has it’s own copy of dagster.yaml. I’ll probably swap to EFS at some point, but just wanted to get some base level of infra up 🙂. it still doesn’t work either way !

Try removing the

image:

line from the Dagster yaml and only setting

DAGSTER_CURRENT_IMAGE=dagster-docker-runner

. Also did you set a volume bind for

/var/run/docker.sock

.

These are my invocation commands to start up each container service:

docker run -v /var/run/docker.sock:/var/run/docker.sock --restart always -d --name dagster-daemon dagster-daemon dagster-daemon run

docker run -v /var/run/docker.sock:/var/run/docker.sock --restart always -d -p 3000:3000 --name dagster-webserver dagster-webserver dagster-webserver -h 0.0.0.0 -p 3000

docker run -v /var/run/docker.sock:/var/run/docker.sock --restart always -d -p 4000:4000 --name dagster-codeserver dagster-codeserver dagster code-server start -f all_code_entry.py --host 0.0.0.0 --port 4000

The

dagster-docker-runner

image definitely exists on the code server:

$ docker images
REPOSITORY              TAG         IMAGE ID       CREATED          SIZE
dagster-docker-runner   latest      e11178232ace   3 seconds ago    360MB
dagster-codeserver      latest      dcdff35e3050   5 minutes ago    361MB
<none>                  <none>      8272e767b9b0   17 minutes ago   361MB
<none>                  <none>      37a4c1a17974   2 hours ago      361MB
<none>                  <none>      6236027e0d24   2 hours ago      361MB
python                  3.12-slim   23cd4350f4bd   6 days ago       130MB

I tried moving

DAGSTER_CURRENT_IMAGE

around to dagster.yaml and each of the service containers (1 x daemon, codeserver, UI webserver). Still getting that login warning. I think dagster is trying to pull a container from the registry instead of locally. That is my guess. Are your containers in dockerhub?

No my containers are local. Try adding

-e DAGSTER_CURRENT_IMAGE='dagster-codeserver'

to the daemon and code-location invocations.

Same issue !

docker.errors.ImageNotFound: 404 Client Error for <http+docker://localhost/v1.41/images/create?tag=latest&fromImage=dagster-codeserver>: Not Found ("pull access denied for dagster-codeserver, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")