https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
m

Mitchell Hynes

12/28/2022, 5:54 PM
Code Locations OSS aren’t as smooth as they are in the Dagster Cloud. I want to add a Daemon to Dagit without editing the workspace.yml directly. Could there be a GraphQL endpoint for this, similar to the one for reloading the workspace? How about a UI for adding locations similar to Code Locations in Cloud, with a reload icon to reload definitions?
d

daniel

12/28/2022, 5:56 PM
Hi Mitchell - I agree that this would be a great improvement to OSS. Is this something you'd be able to file a feature request issue for? We have https://github.com/dagster-io/dagster/issues/6295 but its much more kubernetes-specific
m

Mitchell Hynes

12/28/2022, 5:57 PM
We’re on the same wavelength. I found this issue when I was searching around for Service Discovery features related to this 😆
I’ll make a feature request. Is this too big of a scope for a OSS contribution?
d

daniel

12/28/2022, 5:58 PM
i think it would be a pretty substantial change, yeah
👍 1
m

Mitchell Hynes

12/28/2022, 6:04 PM
Actually r614's suggestion of using an IP/subnet range for service discovery would fit my use case fine.
d

daniel

12/28/2022, 6:05 PM
That's probably something that could be done more quickly than full OSS-cloud parity
although that would be the ideal end-state i think
m

Mitchell Hynes

12/28/2022, 6:09 PM
Okay, thanks Daniel! You seem to be the most active on the repo for issues / Dagit contributions — If I were to request this as a feature, and get a MVP, are PRs like this welcome? What’s the happy path for that?
d

daniel

12/28/2022, 6:09 PM
this is talking about the more scoped service discovery piece now right?
m

Mitchell Hynes

12/28/2022, 6:10 PM
Yes, we use daemons heavily and it would be a huge win to get even that working
d

daniel

12/28/2022, 6:15 PM
There's a contributing guide here: https://docs.dagster.io/community/contributing - generally I think it would be as simple as sending out a PR against the repo. The main potential source of friction to getting it merged that I can imagine is the fact that there would now be k8s-specific logic in the core dagster package - so we would either need to find a way to make the workspace parsing code more pluggable like the dagster.yaml code is, or find some other way to conditionally include kubernetes dependencies The main path for parsing workspace.yaml files and turning them into host/server/port tuples is here: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/workspace/load_target.py#L33-L38
m

Mitchell Hynes

12/28/2022, 6:19 PM
The main potential source of friction to getting it merged that I can imagine is the fact that there would now be k8s-specific logic in the core dagster package
Oh maybe I was being unclear: I don’t use Kubernetes. I use Fargate and EC2 itself. But I don’t think this changes any of your instructions Thanks again. I wanted to get a vibe-check before I dug into it.
d

daniel

12/28/2022, 6:19 PM
Yeah, replace k8s in that case with whatever API you'd be using to query the IP ranges instead - if that's something AWS specific then it would have the same challenge
m

Mitchell Hynes

12/28/2022, 6:50 PM
Assuming a port-scan is out of the question: I looked into Service Discovery since I’m not familiar and it would involve an integration for Consul or even Cloud Map. Yeah, I could see that being a hard sell for obvious reasons. I think I’m going to stick with
scp
the workspace file over + reloading the workspace via GraphQL for now. Unless you have a better idea? I’m fairly new to Cloud Native stuff.
d

daniel

12/28/2022, 6:51 PM
you could have the workspace.yaml specify a module + function to generate the origins
then the module and function could be something you (or e.g. the dagster-aws package) supplies
m

Mitchell Hynes

12/28/2022, 6:55 PM
So the workspace file would have a single entry that multiplexes Definitions. What do you mean by generate the origins in this case?
d

daniel

12/28/2022, 6:55 PM
I meant generate the hostnames and ports to use
m

Mitchell Hynes

12/28/2022, 7:01 PM
Okay I think I’m following! You’d use
python_module
in the workspace to call a function that would generate the hostnames and ports to use, how would you communicate that to Dagit? Is there a Python API I can reference for adding Code Locations to the workspace?
Looking at
load.py
, I think I got my answer! 😄
d

daniel

12/28/2022, 7:25 PM
Yeah, not sure about the specific name but that's the easiest part to change