Code Locations OSS aren’t as smooth as they are in the Dagster Cloud. I want to add a Daemon to Dagit without editing the workspace.yml directly. Could there be a GraphQL endpoint for this, similar to the one for reloading the workspace? How about a UI for adding locations similar to Code Locations in Cloud, with a reload icon to reload definitions?

Hi Mitchell - I agree that this would be a great improvement to OSS. Is this something you'd be able to file a feature request issue for? We have https://github.com/dagster-io/dagster/issues/6295 but its much more kubernetes-specific

We’re on the same wavelength. I found this issue when I was searching around for Service Discovery features related to this 😆

i think it would be a pretty substantial change, yeah

Actually r614's suggestion of using an IP/subnet range for service discovery would fit my use case fine.

That's probably something that could be done more quickly than full OSS-cloud parity

Okay, thanks Daniel! You seem to be the most active on the repo for issues / Dagit contributions — If I were to request this as a feature, and get a MVP, are PRs like this welcome? What’s the happy path for that?

this is talking about the more scoped service discovery piece now right?

Yes, we use daemons heavily and it would be a huge win to get even that working

There's a contributing guide here: https://docs.dagster.io/community/contributing - generally I think it would be as simple as sending out a PR against the repo. The main potential source of friction to getting it merged that I can imagine is the fact that there would now be k8s-specific logic in the core dagster package - so we would either need to find a way to make the workspace parsing code more pluggable like the dagster.yaml code is, or find some other way to conditionally include kubernetes dependencies The main path for parsing workspace.yaml files and turning them into host/server/port tuples is here: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/workspace/load_target.py#L33-L38

The main potential source of friction to getting it merged that I can imagine is the fact that there would now be k8s-specific logic in the core dagster package

Oh maybe I was being unclear: I don’t use Kubernetes. I use Fargate and EC2 itself. But I don’t think this changes any of your instructions Thanks again. I wanted to get a vibe-check before I dug into it.

Yeah, replace k8s in that case with whatever API you'd be using to query the IP ranges instead - if that's something AWS specific then it would have the same challenge

Assuming a port-scan is out of the question: I looked into Service Discovery since I’m not familiar and it would involve an integration for Consul or even Cloud Map. Yeah, I could see that being a hard sell for obvious reasons. I think I’m going to stick with

scp

the workspace file over + reloading the workspace via GraphQL for now. Unless you have a better idea? I’m fairly new to Cloud Native stuff.

you could have the workspace.yaml specify a module + function to generate the origins

So the workspace file would have a single entry that multiplexes Definitions. What do you mean by generate the origins in this case?

I meant generate the hostnames and ports to use

Okay I think I’m following! You’d use

python_module

in the workspace to call a function that would generate the hostnames and ports to use, how would you communicate that to Dagit? Is there a Python API I can reference for adding Code Locations to the workspace?

Yeah, not sure about the specific name but that's the easiest part to change