Hey, I'm following the k8s deployment docs and att...
# announcements
k
Hey, I'm following the k8s deployment docs and attempting to get it running on a k8s server v1.15.12. I commented out all of the
startupProbe
sections in the values.yaml but the helm install doesn't work. It looks like the
templates/deployment-user.yaml
enforces that field here: https://github.com/dagster-io/dagster/blob/master/helm/dagster/templates/deployment-user.yaml#L112-L141. I got it working by pulling it locally and removing that block, but is there a way to get it running without doing this? Or am I missing something?
j
Hi Kirk- I believe the
{{- if $deployment.startupProbe }}
should cause the whole block to be skipped. Are you using the example user code deployment? In that case you would need to explicitly disable it in your values.yaml
Something like
Copy code
userDeployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: latest
        pullPolicy: Always
 
      dagsterApiGrpcArgs:
        - "-f"
        - "/example_project/example_repo/repo.py"
      port: 3030
      startupProbe: {}
helm install --dry-run
could also be helpful for finding any other startupProbes
k
I've commented out everything in the values.yaml relating to
startupProbe
and I get this when it runs:
Copy code
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec.template.spec.containers[0]): unknown field "startupProbe" in io.k8s.api.core.v1.Container
When I pulled the templates dir locally and removed that section I linked above, it all worked. So it definitely seems like it's from the user code example block. I'll test with my own, thanks!
j
If you’re using the example user code (enabled by default), then the startupProbe will be enabled unless you override the settings in userDeployments.deployments in your own values.yaml
Actually- it’s not even necessary to explicitly set the startupProbe to empty as I did above. It’s enough to just provide your own userDeployment entry
Copy code
userDeployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "-f"
        - "/example_project/example_repo/repo.py"
      port: 3030
k
I removed the old and copied that in and ran
helm install dagster dagster/dagster -f values.yaml
and got the same error as above
j
Could you share a file with the result of
helm install --dry-run dagster dagster/dagster -f values.yaml
k
Copy code
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec.template.spec.containers[0]): unknown field "startupProbe" in io.k8s.api.core.v1.Container
j
(Please remove any secrets)
k
That's all it does
j
Gotcha- would you mind sharing your values.yaml? (again please watch out for any secrets)
k
sure
j
Ah I see- the command I was thinking of was
helm template
, which shouldn’t error
k
Do you want that full template yaml output?
j
The values.yaml is probably most useful, but the output could help
k
It should just be the default with the
startupProbe
sections removed
j
Looking in to it. I’m seeing that the dagit and dagster-daemon startupProbes are still enabled with these values.
Likely an issue here with our recommendation to just comment out that lines, since those will fall back to the default. Sorry you’ve bumped in to this
I got it working by pulling it locally and removing that block
@Kirk Stennett to be clear, you only had to remove the probe on userDeployments? When I
helm template
your values, I see probes on dagit and the dagster-daemon deployments
k
Yeah I just removed that block from the deployment-user.yaml file and it works when I run it all locally with
helm install dagster . -f values.yaml
And no worries, I'm new to this and just wanted to make sure I was going about it the right way
j
Hmm this is strange. Could you run
helm template dagster dagster/dagster -f ~/personal-config/test-values.yaml
, and see if you reproduce my result of having 2 startupProbes?
k
It does, I just looked through that after you mentioned it
Oh huh, I added the segment back in and it worked locally too (the deployment-user.yaml segment I removed previously)
j
Did you possibly remove the blocks from
deployment-dagit.yaml
and
deployment-daemon.yaml
? Otherwise it would seem like your install should have failed
k
Now I'm confused as to how that worked
It didn't the first time I ran through, and those blocks still have the:
Copy code
{{- if .Values.dagit.startupProbe }}
          startupProbe:
            {{- toYaml .Values.dagit.startupProbe | nindent 12 }}
        {{- end }}
Is there a different version of the code on github vs. helm? Looks like there was a helm update for dagster, but even with that it still throws the same error
j
Hi Kirk- not sure how this worked the first time. The chart on helm is from the latest dagster release (in this case last friday), and I don’t see any recent changes. This is a bug on our side, it seems disabling the startupProbes isn’t possible in user values.yaml currently. I’ve opened a tracking issue and I’ll look to have a fix out tommorow https://github.com/dagster-io/dagster/issues/3796. In the meantime if you’re able to manually remove the startupProbe blocks from
deployment-dagit.yaml
and
deployment-daemon.yaml
, I think that should get you started. Thanks for your help debugging this!
k
Thanks Johann! I appreciate your help with this