Question concerning deployments. Would the proper pattern (for docker-compose, soon to be kubernetes) docker-compose.yaml, dagster.yaml and .env actually physically residing on the servers for each of the environments and the pipeline code pulled from a docker repository?

Hi george, taking these one by one: • Tou should only need one docker-compose file that is managed outside of dagster and uses in the place where you make the docker-compose file - dagster is not aware that such a file exists • i believe a .env file is read by the docker-compose file but nothing else, so that is only needed in the place you're running docker-compose from • Only dagit and the daemon need a dagster.yaml file

I use docker compose to manage several dagster deployments. I have two docker compose files per deployment: •

docker-compose.yaml

•

docker-compose.build.yaml

The former has everything needed to run containers, assuming that the images have already been built. Both of those files are used when I build the images, i.e.

docker compose --file docker-compose.yaml --file docker-compose.build.yaml build

. This only happens locally or in the cloud (GCP Cloud Build) when the images are being built. I deploy the following files to the place where I want to run dagster: •

dagster.yaml

•

workspace.yaml

•

.env

•

docker-compose.yaml

I can then run

docker compose up

(which relies on

.env

and

docker-compose.yaml

) and the images get fetched from an image repository. I can then run dagster using

docker compose up

(which again relies

.env

and

docker-compose.yaml

) which starts the containers, and the containers then bind mount

dagster.yaml

and

workspace.yaml

. I could simplify this by storing building

dagster.yaml

and

workspace.yaml

into the docker images and may move towards that in the future. @daniel is right in what he says about the separation between dagster files (

dagster.yaml

and

workspace.yaml

) and docker files (

.env

and

docker-compose.yaml

).