I am going to ask in here but might not be the best place. We have a scripting server that runs all our python code but I want to run the dagit web interface and scheduler on a different machine. How do I send the commands over to the scripting server to run the jobs. I see something bout the gRPC server and I am working setting this up in a docker on my monitoring machine.
02/28/2023, 4:55 PM
Hey Joe - can you clarify what you're referring to by "How do I send the commands over to the scripting server to run the jobs."? Maybe an example?
We have a graphql api that lets you tell Dagster to launch runs over an aPI: https://docs.dagster.io/concepts/dagit/graphql#graphql-api - but that might be different that what you're asking for here
03/01/2023, 6:28 PM
sorry I probably worded that wrong. I am trying to do an on-prem setup. I have dagster running in a docker and I have my other python scripts on a different server in dockers themselves. I am trying to figure out how to call those scripts in the other dockers. my guess is I need to use the RPC to make those calls. I have been following the documentation but everything shows from the same machine nothing shows combining multiple machines together. I hope this makes more sense.
03/01/2023, 6:30 PM
Got it - i don't think dagster has any specific abstractions to help with making RPC calls to other containers, but I also wouldn't expect it to be any harder than it would be if you weren't using dagster
i.e. if you have some way of doing it in Python, you can call that within your Dagster op and I think it would still work