A very similar question here: what’s the go-to deployment model when dealing with multiple teams? Do you centralize the dagit instance and let teams deploy their pipelines against that central unit? Do you let each teach deploy everything for each project? How do you deal with the repositories being managed by another team?

Hey @Huib - the recommended deployment model is a single dagit instance, that each team deploys their own repositories against

Thanks! How does that work in practice? All the examples I’ve seen have the available repositories in the deployment of the dagit instance

@daniel - is this something you’d be able to answer? The depth of my knowledge is limited here

Hi! we have an example here that demonstrates a separate deployment for pipeline code that could be extended in a striaghtforward way to have multiple deployments for different teams https://docs.dagster.io/deployment/guides/docker#example

@daniel My current understanding is that the deployment flow would look something like this: A central team has a git repo containing the dagit + dagster code, keeping all the pipeline repositories in one file. Each team wanting to schedule something creates their own pipelines, and runs their own grpc server After deploying the grpc server to k8s, the team submits a pull request to the workspace.yaml file on the central git repo, requesting their new pipeline to be included in the central tool. Does this sound roughly correct?

That sounds largely right to me, with the possible exception of each team needing to run its own gRPC server. If you're in k8s you might find this section interesting: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#separately-deploying-dagster-infrastructure-and-user-code

After deploying the grpc server to k8s, the team submits a pull request to the workspace.yaml file on the central git repo, requesting their new pipeline to be included in the central tool. Does this sound roughly correct?

Just to be clear, this should only need to happen once per team, when that team onboards, not every time that the team pushes a change or adds a new pipeline, right?

From what I understood, yes, you would deploy the grpc server per team, which can serve multiple projects

Having each team deploy their own gRPC server is definitely an option. It's not currently the approach we take in our example helm chart, but the system is set up a way that would allow it

oh wait, in your example, would you run the grpc server centralized?

our example helm chart runs all the gRPC servers for you (based on images that could be built and tagged by each team individually using their own build and deployment process)

and just for giggles: I could do something like

image_name:latest

right? and just click the refresh button in the UI whenever I made a small change to the DAG?

you could use latest tags, yeah - we don't necessarily recommend that it might cause older queued runs to launch using the new code, which won't always be compatible

Ah yeah. Have you run dagster in larger corporations in a centralized manner? What were the biggest pain points?

Yeah, we have a few case studies that go into more depth about their deployment setups in production. For example, https://dagster.io/blog/shared-spaces-at-drizly - I'm not sure how deep this goes into the user code deployment side of things specifically though