My repo and pipeline definitions in `repo py` seem to be fin dagster #ask-community

My repo and pipeline definitions in `repo.py` seem...

Scott Peters

06/21/2021, 2:05 AM

My repo and pipeline definitions in

repo.py

seem to be fine, as

dagster

doesn't complain about them when running

docker-compose -f docker-compose.yml up

, but I must be missing something about how the mapping between the

workspace.yaml

and where

dagster

finds the docker images / repos? Currently, the

repo.py

for

beekeeper_pipelines

is copied over to

/opt/dagster/app

in it's container, and the

workdir

is set appropiately Also, I'm assuming that adding new containers to the running

dagit

dagster daemon

is as simple as starting the

new_service

container with the proper args and joining the proper docker network? ( without having to add the new service to this example compose file, rebuild and up? )

Copy code

- keep running instance of dagster/dagit
- pull new_repo container
- start new_repo container on the dagster docker network
- add new_repo container to workspace.yaml
- refresh repos within dagit

??

daniel

06/21/2021, 2:00 PM

Hi Scott - this should work the way you want if you make one small change to the Dockerfile in the example. Specifically it needs to mount the workspace.yaml file as a volume file rather than copying it in when the Dockerfile is built. That way you can make changes to the workspace.yaml file without needing to rebuild the container We should probably make that change to the example - in the meantime you would need to rebuild the Dockerfile and restart dagit in order for additions to the workspace.yaml to show up there

Scott Peters

06/21/2021, 8:27 PM

for sure, I figured we could keep the

workspace.yaml

live on the server and simply pipe it in like you said via volume mount.

Scott Peters

06/21/2021, 8:31 PM

That been said, I am still trying to workout how

dagster

finds the repos via

docker

. Is there any assumed naming on

dagster's

part? For example, in the

docker_example_pipelines

, is there anything magical about that naming? Do I need to append my image_name, or repo with

docker

? referring to the last image, it seems that

dagit/dagster

properly sees the

beekeeper

container and instantiates it correctly, but the repo doesn't show up in dagit? ... is there a way to view the

workspace.yaml

in a raw format in

dagit

? perhaps that way I can be sure it isn't caching anything... or some way of getting

dagster

to bail / error out when a repo entry in the

workspace.yaml

doesn't fully load into the system?

daniel

06/21/2021, 11:27 PM

The way that dagster finds the repos is that each service exposes a gRPC port and hostname (using docker networking), and the workspace.yaml file tell dagit which host/port to look at. Re: "is there a way to view the

workspace.yaml

in a raw format in

dagit"

- The "Workspace" page you posted in your first set of screenshots is more or less this, which makes me think dagit is not reading your updated

workspace.yaml

. After you rebuild and restart the dagit container (the rebuild step is key given that it's not mounting the volume) is it still only showing one location? That would be surprising

Scott Peters

06/22/2021, 1:44 AM

Hey Daniel, yeah, it the

workspace.yaml

was not being piped in, so that solves that issue.

93 Views

Open in Slack

Previous Next