Hi, fairly new user here. Ran dagster+dagit locally, and now we want to adopt it on a broader scale, so I have a couple of general questions/wondering about best practices. Our idea is to deploy to ECS and spawn user tasks as needed, but willing to switch to another deployment scheme if need be. As I understand I have to tell dagit where the user code lives, either via workspace.yaml or by specifying command line arguments upon start. I was wondering what the scalable solution is to having many user code repositories. Do I have to update the workspace file every time that I add a new repository (say, load_from a new grpc connection)? Or is there a way for user agents to reach out to dagit and tell it about themselves? What I want to do is enable users to develop/add repositories on their own, while I IT/ops makes sure that dagit is up and running. Once the users are happy with their code, they push their image. I am wondering whether it's possible for that image to be started on ECS/k8s/... and connect to the long-running dagit instance, instead of extending the dagit's instance workspace.yaml with the new user code location (effectively requiring a restart?). Am I remotely on the right track here, or did I get some concepts completely backwards? Many thanks in advance!

Hi Robert - we have a feature request issue for moving to more of a service discovery model for the grpc servers: https://github.com/dagster-io/dagster/issues/6295, but until that's built there does need to be a central workspace.yaml, yeah (although you don't need to restart the service when it changes - you can mount it as a volume and then reload the contents of the file within dagit) For what it's worth the cloud product does work a more like that today (although we’re not holding it back for the cloud product, I suspect dagster open source will eventually have this property as well) - you can spin up a new code location in dagit or via our API and then the agent will spin everything up for you without needing to restart anything

Thanks, that's good news! I'll be watching the issue closely 🙂

@Robert Schmidtke thank you for bringing this up. I -- a user just like you -- think the repository discovery feature is a sorely needed feature to truly make dagster an "enterprise" tool that can be used not only to orchestrate jobs but also the team that works on them.