https://dagster.io/ logo
x

Xu Zhang

03/06/2021, 2:31 PM
Guys, I wanted to have one or more gRPC servers that provide repositories. I understand I need to make the workspace.yaml looks like the one below: load_from: - grpc_server: host: localhost port: 4266 location_name: 'my_grpc_server My question is: how do I dynamically add new gRPC servers without rebooting Dagster instance? Is there a lazy loading function that could return a list of gRPC servers?
d

daniel

03/06/2021, 3:41 PM
Hi Xu - not currently but this is on our immediate roadmap. We should likely be able to support this in dagit as part of the upcoming 0.11.0 release in just a few weeks. (So you would be able to modify workspace.yaml and then be able to reload the workspace from within dagit without restarting it)
x

Xu Zhang

03/06/2021, 4:03 PM
ack ty daniel
a follow up question: is that possible to construct pipelines using solids from different repos from different gRPC server?
n

Noah K

03/07/2021, 12:10 PM
Kind of, watch my preso for the solution I've been using, basically the pipeline is built using blocking proxy objects with the actual logic in another daemon
x

Xu Zhang

03/07/2021, 7:12 PM
@Noah K do you have the link to you preso?
@daniel regarding the followup question ^^
d

daniel

03/08/2021, 8:00 PM
That's something we'd like to enable in the future with some changes to the way the orchestrator process works, but isn't currently possible.
x

Xu Zhang

03/08/2021, 8:02 PM
ack
r

rex

03/08/2021, 8:08 PM
@Xu Zhang link to noah’s talk:

https://www.youtube.com/watch?v=1buRROJ3m2s

x

Xu Zhang

03/08/2021, 8:08 PM
ty!