Hi, What is the best way to debug a dagster gRPC s...
# announcements
l
Hi, What is the best way to debug a dagster gRPC server that I've kicked using the API, following this example https://github.com/dagster-io/dagster/tree/0.11.0/examples/deploy_docker ? The gRPC server container is not logging anything as far as I can tell, while
dagster api grpc-health-check
returns nothing. The error I am getting in dagit is:
Copy code
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1616415994.552532757","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5396,"referenced_errors":[{"created":"@1616415994.552531220","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"
Thanks for your help
d
Hi Lewis - typically if the gRPC server runs into an error while starting up there should be some kind of error. It’s also possible that the configuration in your workspace.yaml isn’t pointing at the right host or port - if you post your workspace file we could take a look. Do you run into this error when you run the example out of the box?
l
Hi Daniel, here is my workspace.yaml
Copy code
load_from:
  - grpc_server:
      host: localhost
      port: 4000
      location_name: "pipelines"
I've taken parts from the example and applied them to my repo
Do you know how I would best go about seeing error for the gRPC server? I am using docker-compose to launch the containers. If the gRPC server is logging to
STDERR
or
STDOUT
then it should be picked up by docker but I can't see anything
The server is being launched with this:
Copy code
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repos.py"]
d
Yeah, typically if there's an error in the gRPC server it should be there in the container logs. Is your docker-compose file pretty similar to the example?
As a sanity check, one thing you could do is verify whether the example with no local changes works for you - that would help suss out if the problem has something to do with your environment (as opposed to one of the changes you made)
l
Yeah it's pretty similar to the example
I am also getting a response when I go to localhost:4000 so it clearly is running to some extent, but the response is just a string of seemingly random characters. Looks like an encoding issue
I'll give that a go
d
Do you get the same error if you go to reload the repository locaiton in dagit? On the status page there's a reload link
one thing that jumps out is that your host is 'localhost' but the example uses a docker-provided hostname - could that have something to do with the connection issue?
l
That was it, thanks so much for your help!
d
np! I think we should make the gRPC server not expose port 4000 to localhost in the example, that's confusing since it's not what dagit uses. Will make that change!
l
Is there any documentation on how grpc_sever host is resolved from workspace.yaml?
Because I'm looking to deploy this on AWS using EC2s and I'm not sure how the name of the docker container will carry over to that environment
d
that hostname is set by docker-compose (it's the name of the docker-compose service). Is your AWS setup also using docker-compose?
@Dagster Bot issue Missing user code deployment example in AWS
d
l
No it wouldn't be using docker-compose. I'm effectively managing that myself, launching each container separately. At the moment I'm thinking of hosting the gRPC server, exposing it at say https://my-domain/dagster-grcp . Would I then be able to specify a workspace.yaml of :
Copy code
load_from:
  - grpc_server:
      host: <https://my-domain/dagster-grcp>
      location_name: "pipelines"
?
d
I'd expect that work although you shouldn't need the 'https://'
l
On that basis, do you know why it doesn't work when I provide this workspace.yaml:
Copy code
load_from:
  - grpc_server:
      host: localhost
      port: 4000
      location_name: "pipelines"
I'm just hoping to validate that it'll work locally before I move the build into AWS. Is there any code/docs that you can point me to about the dagster grpc_server? I just can't find anything online to inform me about what's going on behind the scenes Thanks again for all your help on this
d
np! I think it's because your dagit doesn't have access to localhost since it's running in docker / localhost doesn't resolve to the same place within that container as it does in your browser
l
Oh okay, that makes sense. I tried the config from here https://dagster.slack.com/archives/CCCR6P2UR/p1616425831143900?thread_ts=1616416315.128200&amp;cid=CCCR6P2UR and got the following error:
dagster.check.CheckError: Invariant failed. Description: must supply either a socket or a port
. Not sure how I'd get around that or if supplying 80/443 for http/https would work
d
Yeah, it needs you to specify the port on which the server is listening - that can be 80 if your server is set up to be listening on that port.
an example AWS setup with user code deployments would help a lot here I think - filed an issue to include that in the docs
l
Thanks Daniel, that's helped a lot
269 Views