https://dagster.io/ logo
Title
r

rowan gaffney

03/01/2022, 11:27 PM
Is it possible to have multiple workspaces/repositories per user-code-deployment?
e.g. the equivalent of https://docs.dagster.io/tutorial/advanced-tutorial/repositories#supporting-multiple-repositories, but from the grpc in the user-code-deployment helm chart
c

chris

03/02/2022, 12:00 AM
yes - you can have multiple repositories in a given user code deployment. When you specify the repository location, all repositories defined within the module will be pulled in.
r

rowan gaffney

03/02/2022, 4:44 PM
New to Dagster, so I wonder if I am getting the terminology/concepts wrong... What I'd like to do is if I have: |-project1 |----jobs |----ops |----project1_repo.py |-project2 |----jobs |----ops |----project2_repo.py Where each project is a workspace. My understanding is that this can be achieved via the workspace.yaml like:
load_from:
  - python_file: /project1/project1_repo.py
  - python_file: /project2/project2_repo.py
What would the syntax for this look like in the user-code-deployment chart?
c

chris

03/02/2022, 10:51 PM
Have you checked out https://docs.dagster.io/concepts/repositories-workspaces/workspaces#running-your-own-grpc-server? You can point the grpc server at a particular repo file and attribute, and then specify the config in your workspace.yaml
b

Baris Cekic

08/23/2022, 2:59 PM
hey @chris and @rowan gaffney, same interest here. Have you find a solution for this? I dont understand how my own grpc server helps in this context?
c

chris

08/23/2022, 9:54 PM
You can configure
dagster-user-deployments
in the helm chart directly. Each user code deployment corresponds to a grpc server that's communicating back with the dagit webserver: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#deploying-dagster-on-helm check out the
dagster-user-deployments
section on the helm chart
b

Baris Cekic

08/23/2022, 10:26 PM
hey @chris, thanks for the info. I am new to Dagster and love it so far. I am trying to replace our Airflow in Kubernetes. I understand the
dagster-user-deployments
but it is hard to manage multiple repositories and frequent code changes. I have to redeploy the helm chart for every repo and corresponding gRpc or I need to redeploy the user-deployment. In our use case, there are multiple users , frequently create repos, make code changes. I am trying to find a way to automatically syncronize git repos and reflect code changes.
Is there any other way to manage multiple repositories and code changes other than redeployment or how can I prevent downtimes in my use-case? I believe you have production customers on multiple repo/frequent code changes.
c

chris

08/23/2022, 10:41 PM
hey baris - valid question - I'm a bit out of my depth to answer that, so going to get a team member with more experience with helm/k8s to get back to you about this
b

Baris Cekic

08/23/2022, 10:46 PM
thanks @chris
j

johann

08/24/2022, 1:03 PM
Hi Baris, most users have a CICD pipeline that runs the deploy step (helm upgrade/whatever platform they’re on) on each commit
It’s worth mentioning that Cloud has one of these CICD pipelines out of the box https://dagster.io/cloud
b

Baris Cekic

08/24/2022, 1:05 PM
But helm deployment will have a downtime on dagit / user deployment , right?
j

johann

08/24/2022, 1:06 PM
It shouldn’t. If you just updated your user code, your dagit deployment won’t need to change at all. And the user deployment can do a rolling upgrade
Dagster is also mostly tolerant of blips of downtime from the user deployment, e.g. any schedules that tried to use it will just pick back up once it can be reached
b

Baris Cekic

08/24/2022, 5:52 PM
what about multiple repositories? Do you recommend use same user deployment for multiple repositories or different user-deployments per github repo/branch? In our use-case there will be multiple users and multiple git repos and multiple branches. So it doesn’t look efficient to spin up a user deployment for each repo and branch. What are your recommendations?
j

johann

08/24/2022, 8:14 PM
If deploying to k8s, a user deployment is essentially 1 to 1 with whatever code you want to package in to a docker image. It’s certainly easiest to have 1 per branch
Last time I’ll shill for it haha, but cloud has a lot of prebuilt features that are oriented around having multiple teams pushing to different branches/repos. Could take a look at https://docs.dagster.io/dagster-cloud/developing-testing/branch-deployments