Guys, I wanted to have one or more gRPC servers that provide repositories. I understand I need to make the workspace.yaml looks like the one below:
load_from:
- grpc_server:
host: localhost
port: 4266
location_name: 'my_grpc_server
My question is: how do I dynamically add new gRPC servers without rebooting Dagster instance?
Is there a lazy loading function that could return a list of gRPC servers?
d
daniel
03/06/2021, 3:41 PM
Hi Xu - not currently but this is on our immediate roadmap. We should likely be able to support this in dagit as part of the upcoming 0.11.0 release in just a few weeks. (So you would be able to modify workspace.yaml and then be able to reload the workspace from within dagit without restarting it)
x
Xu Zhang
03/06/2021, 4:03 PM
ack ty daniel
a follow up question: is that possible to construct pipelines using solids from different repos from different gRPC server?
n
Noah K
03/07/2021, 12:10 PM
Kind of, watch my preso for the solution I've been using, basically the pipeline is built using blocking proxy objects with the actual logic in another daemon
x
Xu Zhang
03/07/2021, 7:12 PM
@Noah K do you have the link to you preso?
@daniel regarding the followup question ^^
d
daniel
03/08/2021, 8:00 PM
That's something we'd like to enable in the future with some changes to the way the orchestrator process works, but isn't currently possible.