https://dagster.io/ logo
Title
s

Scott Peters

06/21/2021, 2:05 AM
My repo and pipeline definitions in
repo.py
seem to be fine, as
dagster
doesn't complain about them when running
docker-compose -f docker-compose.yml up
, but I must be missing something about how the mapping between the
workspace.yaml
and where
dagster
finds the docker images / repos? Currently, the
repo.py
for
beekeeper_pipelines
is copied over to
/opt/dagster/app
in it's container, and the
workdir
is set appropiately Also, I'm assuming that adding new containers to the running
dagit
/
dagster daemon
is as simple as starting the
new_service
container with the proper args and joining the proper docker network? ( without having to add the new service to this example compose file, rebuild and up? )
- keep running instance of dagster/dagit
- pull new_repo container
- start new_repo container on the dagster docker network
- add new_repo container to workspace.yaml
- refresh repos within dagit

??
d

daniel

06/21/2021, 2:00 PM
Hi Scott - this should work the way you want if you make one small change to the Dockerfile in the example. Specifically it needs to mount the workspace.yaml file as a volume file rather than copying it in when the Dockerfile is built. That way you can make changes to the workspace.yaml file without needing to rebuild the container We should probably make that change to the example - in the meantime you would need to rebuild the Dockerfile and restart dagit in order for additions to the workspace.yaml to show up there
s

Scott Peters

06/21/2021, 8:27 PM
for sure, I figured we could keep the
workspace.yaml
live on the server and simply pipe it in like you said via volume mount.
That been said, I am still trying to workout how
dagster
finds the repos via
docker
. Is there any assumed naming on
dagster's
part? For example, in the
docker_example_pipelines
, is there anything magical about that naming? Do I need to append my image_name, or repo with
docker
? referring to the last image, it seems that
dagit/dagster
properly sees the
beekeeper
container and instantiates it correctly, but the repo doesn't show up in dagit? ... is there a way to view the
workspace.yaml
in a raw format in
dagit
? perhaps that way I can be sure it isn't caching anything... or some way of getting
dagster
to bail / error out when a repo entry in the
workspace.yaml
doesn't fully load into the system?
d

daniel

06/21/2021, 11:27 PM
The way that dagster finds the repos is that each service exposes a gRPC port and hostname (using docker networking), and the workspace.yaml file tell dagit which host/port to look at. Re: "is there a way to view the 
workspace.yaml
 in a raw format in 
dagit"
- The "Workspace" page you posted in your first set of screenshots is more or less this, which makes me think dagit is not reading your updated
workspace.yaml
. After you rebuild and restart the dagit container (the rebuild step is key given that it's not mounting the volume) is it still only showing one location? That would be surprising
s

Scott Peters

06/22/2021, 1:44 AM
Hey Daniel, yeah, it the
workspace.yaml
was not being piped in, so that solves that issue.