Robert Schmidtke

03/04/2022, 12:36 PM
Hi, fairly new user here. Ran dagster+dagit locally, and now we want to adopt it on a broader scale, so I have a couple of general questions/wondering about best practices. Our idea is to deploy to ECS and spawn user tasks as needed, but willing to switch to another deployment scheme if need be. As I understand I have to tell dagit where the user code lives, either via workspace.yaml or by specifying command line arguments upon start. I was wondering what the scalable solution is to having many user code repositories. Do I have to update the workspace file every time that I add a new repository (say, load_from a new grpc connection)? Or is there a way for user agents to reach out to dagit and tell it about themselves? What I want to do is enable users to develop/add repositories on their own, while I IT/ops makes sure that dagit is up and running. Once the users are happy with their code, they push their image. I am wondering whether it's possible for that image to be started on ECS/k8s/... and connect to the long-running dagit instance, instead of extending the dagit's instance workspace.yaml with the new user code location (effectively requiring a restart?). Am I remotely on the right track here, or did I get some concepts completely backwards? Many thanks in advance!
👍 2


03/04/2022, 1:13 PM
Hi Robert - we have a feature request issue for moving to more of a service discovery model for the grpc servers:, but until that's built there does need to be a central workspace.yaml, yeah (although you don't need to restart the service when it changes - you can mount it as a volume and then reload the contents of the file within dagit) For what it's worth the cloud product does work a more like that today (although we’re not holding it back for the cloud product, I suspect dagster open source will eventually have this property as well) - you can spin up a new code location in dagit or via our API and then the agent will spin everything up for you without needing to restart anything
👍 1

Robert Schmidtke

03/04/2022, 1:16 PM
Thanks, that's good news! I'll be watching the issue closely 🙂

Will Gunadi

03/05/2022, 12:03 AM
@Robert Schmidtke thank you for bringing this up. I -- a user just like you -- think the repository discovery feature is a sorely needed feature to truly make dagster an "enterprise" tool that can be used not only to orchestrate jobs but also the team that works on them.