Allowing for a centralized instance that teams can (decentrally) use for their projects would be amazing, and it would allow a setup similar to prometheus, where no change is needed to configure new endpoints

Hi @Huib Keemink - this is a cool idea that wouldn't be a huge change on top of the existing system - the part of the system that expects a single centralized

workspace.yaml

file is here and would be reasonably pluggable: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/core/workspace/load_target.py#L27 I could imagine a different WorkspaceLoadTarget that used the k8s python API to do service discovery instead, for example. For now I filed an issue here to track this: https://github.com/dagster-io/dagster/issues/6295

Thanks, really hope I’m not the only who would find this useful! 🙂

Here I propose a workaround that works very well for us https://dagster.slack.com/archives/C01NXMEPJ7R/p1657807447451829