Josh Lloyd
05/19/2023, 1:55 AMhelm install
and get all 4 pods (dagit, daemon, user-code, postgres) standing up without error from k8s’ perspective. When I open the dagit UI however the status of the user-code location has that oh so annoying gRPC Error code: UNAVAILABLE
error message.
When I do a kubectl logs <user_code_pod>
I get no logs returned, not a single log line. This is odd because I thought I’d at least see something like Started Dagster code server for file /example_project/example_repo/repo.py on port 3030 in process 1
. In a separate isolated docker container I have proven that the dagster api grpc
command is working and will kick of the server with the expected message above, but I don’t see it in my helm deployment.
My values.yaml
file is minimally altered from the k8s deployment docs example — only enough to add some secrets and point to my local custom user-code image.
I have no idea where to look next to debug. Any thoughts?jordan
05/19/2023, 4:10 PMJosh Lloyd
05/19/2023, 4:14 PMkubectl describe pod
has not produced any other messages that seem to help other than confirming that it’s start up command is correct and that it just actually spin up without error.Readiness probe failed: Get "<http://172.17.0.9:80/dagit_info>": dial tcp 172.17.0.9:80: connect: connection refused
kubectl
context. It was similar but in a different namespace, so I wonder if dagit was trying to connect to a pod that didn’t exist based on the misconfigured namespace. All I had to do was switch to the right context and it started working again.