https://dagster.io/ logo
Powered by
Title
m

Mohammad Nazeeruddin

10/12/2021, 12:38 PM
Hi Team. We Deployed user-code-deployment  with dagster official helm chart but we are getting below error in dagit : 
rpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE details = "no healthy upstream" debug_error_string = "{"created":"@1634031954.613875382","description":"Error received from peer ipv4:192.168.125.148:3030","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"no healthy upstream","grpc_status":14}" This our workspace.yaml and gateway yaml files in helm. in gateway we configured Istio things to access our dagit ui. Please help us to resolve this error.
n

Nilesh Pandey

10/12/2021, 3:06 PM
Does anyone any suggestions for this ?
j

johann

10/12/2021, 3:43 PM
Dagit is showing that error on the workspace page for your repo location?
r

rex

10/12/2021, 4:41 PM
Are you using just the 
dagster
Helm chart, or both 
dagster
and 
dagster-user-deployments
? Could you also send us the Helm values that you’re using?
m

Mohammad Nazeeruddin

10/13/2021, 5:17 AM
@rex Yes we are using dagster Helm chart and user-code-deployments and i am sharing our values file.
{{- $userDeployments := index .Values "dagster-user-deployments" }}
{{- if and (not $userDeployments.enabled) $userDeployments.enableSubchart }}
{{ fail "dagster-user-deployments subchart cannot be enabled if workspace.yaml is not created." }}
{{- end }}
{{- if $userDeployments.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
  
name: {{ template "dagster.fullname" . }}-workspace-yaml
  
labels:
    
app: {{ template "dagster.name" . }}
    
chart: {{ template "dagster.chart" . }}
    
release: {{ .Release.Name }}
    
heritage: {{ .Release.Service }}
data:
  
workspace.yaml: |
    
{{- $dagitWorkspace := .Values.dagit.workspace }}
    
{{- $deployments := ternary $dagitWorkspace.servers $userDeployments.deployments $dagitWorkspace.enabled }}
    
{{- if $deployments }}
    
load_from:
      
{{- range $deployment := $deployments }}
      
{{- $deploymentHost := ternary $deployment.host $deployment.name $dagitWorkspace.enabled }}
      
- grpc_server:
          
host: {{ $deploymentHost }}
          
port: {{ $deployment.port }}
          
location_name: {{ $deploymentHost }}
      
{{- end }}
    
{{- else }}
    
load_from: []
    
{{- end }}
{{- end }}
this is configmap-workspace.yaml file ^
i logged dagit pod and in worksapce.yaml host , location_name same. tried with my repo_name called example_repo but same error i got. i was confused between these two host and location_name.
user-code-deployment file.
deployment-user.yaml
docker-compose.yaml file is required ?
r

rex

10/13/2021, 6:03 AM
For each user code deployment, a service is created to expose access. Dagit communicates to this server using gRPC. In the workspace.yaml, the host is the name of this created service. Are the user code services also configured in your Istio service mesh? Otherwise, it may not be letting dagit communicate if the user code service is considered as an external service to the mesh.
docker-compose.yaml file is required ?
No, this Helm deployment works out of the box.
👍 1
m

Mohammad Nazeeruddin

10/13/2021, 6:09 AM
In the workspace.yaml, the host is the name of this created service. > yes the user-code-deployment service name but location-name also same so is it correct? so there is np with workspace.yaml?
this is our istio setup file SS i was added dagster-dagit service so i need to add user-code service name too. ? so then Grpc issue resolved right?
r

rex

10/13/2021, 6:21 AM
I’m not familiar with Istio. But if you configured your service mesh so that you can’t access external services, then I would assume grpc would break until the user code services are in the mesh.
m

Mohammad Nazeeruddin

10/13/2021, 6:27 AM
okay Thank you @rex.
Hi @rex. user code service in istio But getting below error : so if i configured user code service in istio ,Grpc server won't work? 
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE details = "upstream connect error or disconnect/reset before headers. reset reason: protocol error" debug_error_string = "{"created":"@1634211334.767575227","description":"Error received from peer ipv4:192.168.238.41:3030","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"upstream connect error or disconnect/reset before headers. reset reason: protocol error","grpc_status":14} How i can solve this grpc issue any suggestion?
any suggestion please ? to solve this
n

Nilesh Pandey

10/18/2021, 11:34 AM
This is blocker for us right now
m

max

10/18/2021, 3:16 PM
it's hard to tell what exactly is wrong here but googling around for that grpc error surfaces a bunch of different ways that istio services can be misconfigured
some things i would double check include https termination, as well as whether anything is traveling over http2
looks like a bunch of people hit this kind of thing when there's a service name mismatch as well
it might help if you shared your new istio config file
would also be helpful to see debug logs from the proxyt
n

Nilesh Pandey

10/18/2021, 3:19 PM
@sainimmakayala
d

daniel

12/07/2021, 6:10 PM
Hey Mohammed - this post is a couple months old, but do you remember how you were able to resolve this? Another user is reporting a similar issue and just curious what you uncovered
m

Mohammad Nazeeruddin

04/08/2022, 9:38 AM
type: "ClusterIP"
  ports:
    - name: grpc   # added grpc instead of http.
      port: {{ $deployment.port }}
      protocol: TCP
      targetPort: 3030   # grpc port
In user-code-deployment subchart file service-user.yaml
#dagster-supportJoin Slack
Powered by