Hey all, I am working on the design for our user-deployment-code for our k8 dagster instance and I am trying to find the right design for repositories..... We have multiple teams who will be looking to create jobs and manage assets and I can see the benefits of them having each their own repository, however, there is also the desire for shared assets used by all teams. One idea was to package the shared assets separately, allowing users to install them as dependencies for their own repos / container deployments. However, I feel that this might impose too much-working knowledge as a barrier to entry for new teams/understanding all the infra pieces for dagster. The other is to create a single container with everyone's project and deploy repositories using their team's repo path:

- project_1
  repo.py
- project_2
  repo.py
- shared
  some_shared_resource

and then use the same container image for each deployment repo, allowing them to directly reference the shared package from their code like they would locally.

deployments:
  -name: "project_1"
   image:
     repository: "same_image"
   dagsterApiGrpcArgs:
     - "project_1/repo.py"
  -name: "project_2"
   image:
     repository: "same_image"
   dagsterApiGrpcArgs:
     - "project_2/repo.py"

Wondering if anyone has good experience with this or examples of what their orgs did to manage multiple teams. Or if they simply found it easier to create one repo for all. The draw to multiple repos is more so simply a way of containing blast radius if one team causes a bad deployment, making it so that not all teams are affected. Other then that I don't see many examples of teams wanting to spin up different versions of python or different working environments for their code.

I am working on the same problem but probably next week. Given that I worked with fluxcd and gitops. My plan is to have hard backed docker images but create branch images which pull in the latest code changes straight from the git repo without any docker image rebuild. Insides the branches the teams can have as many @repositories as they want. Then using git-sync to pull in the code. That saves the roundtrip of building the code image first, then updating in argocd. A very basic version would be to parametrize the image and not put the code in but pull it on startup. With your super repo setup I see the issue that if any project commits the docker image has to be rebuilt/pulled/restarted. But the standard idea would be I believe to manage it in separate helm chart user-deployments. update version -> rollout … then you can use helm to rollback and so on. Then your teams/projects can easily manage their own (code) helm charts and docker builds. This does not require much know how, if someone else just hands you the chart + some basic commands. I think for reliable rollbacks and deployments I would go helm. If you want to enable individual data scientist to get their code online in seconds then my gitops sync. The super repo would work if you don’t think there is too many commits on master/staging branches per day (depends on team size and CD I guess).

@Samuel Stütz So just to summarize you're creating images per team, however just having those images pull in teams code from the docker image itself? Does this impact traceability for what code is being used within the container? I believe that dagster runs reference with the image the pod is using so there are benefits of new versions of the container being deployed (unless I am missing something where there is a disadvantage with containers being rebuilt in dagster?)

Yes you could say team. To be honest haven tested it yet. MLflow logs the githash and not the container but yes you are right the run (not code) container would also need to fix its code be rebuilt and there it would make more sense just for speed alone to have it built fully. Dagster doesn’t build anything as I understand it. You build the code container and potentially a different lighter run(executor) container or use the same for both as in the example. The run image I think can be specified for the executor entirely separate of the code. So I can just build a run image locally push tag it push it and then run it while only configuring some sort of pipeline logic + which container to run in the code itself. The thing with the code deployments is that it may require updating the image in the helm/kustomize. Thinking about this some more I guess a code-deployment-image:{team}-{stage} with pull always and k8s deployment restart. Then have run-images built in a separate script but I would keep this closer to the actual code. In general I still would give the run-image build scripts as close to the actual code as possible. Then manage the code-images + deployment with helm or argocd by the teams themselves. Dagster is just infrastructure then.