Hey I m following the k8s deployment docs and attempting to

Question

Hey I m following the k8s deployment docs and attempting to get it running on a k8s server v1 15 12 I commented out all of the `startupProbe` sections in the values yaml but the helm install doesn t work It looks like the `templates deployment user yaml` enforces that field here <https github com dagster io dagster blob master helm dagster templates deployment user yaml L112 L141> I got it working by pulling it locally and removing that block but is there a way to get it running without doing this Or am I missing something

johann · Answer

Hi Kirk I believe the ` if $deployment startupProbe ` should cause the whole block to be skipped Are you using the example user code deployment In that case you would need to explicitly disable it in your values yaml

johann · Answer

Something like ```userDeployments enabled true deployments name k8s example user code 1 image repository <http docker io dagster user code example|docker io dagster user code example> tag latest pullPolicy Always dagsterApiGrpcArgs f example project example repo repo py port 3030 startupProbe ```

johann · Answer

`helm install dry run` could also be helpful for finding any other startupProbes

Kirk Stennett · Answer

I ve commented out everything in the values yaml relating to `startupProbe` and I get this when it runs ```Error unable to build kubernetes objects from release manifest error validating error validating data ValidationError Deployment spec template spec containers 0 unknown field startupProbe in io k8s api core v1 Container``` When I pulled the templates dir locally and removed that section I linked above it all worked So it definitely seems like it s from the user code example block I ll test with my own thanks

johann · Answer

If you re using the example user code enabled by default then the startupProbe will be enabled unless you override the settings in userDeployments deployments in your own values yaml

johann · Answer

Actually it s not even necessary to explicitly set the startupProbe to empty as I did above It s enough to just provide your own userDeployment entry ```userDeployments enabled true deployments name k8s example user code 1 image repository <http docker io dagster user code example|docker io dagster user code example> tag latest pullPolicy Always dagsterApiGrpcArgs f example project example repo repo py port 3030```

Kirk Stennett · Answer

I removed the old and copied that in and ran `helm install dagster dagster dagster f values yaml` and got the same error as above

johann · Answer

Could you share a file with the result of `helm install dry run dagster dagster dagster f values yaml`

Kirk Stennett · Answer

```Error unable to build kubernetes objects from release manifest error validating error validating data ValidationError Deployment spec template spec containers 0 unknown field startupProbe in io k8s api core v1 Container```

johann · Answer

Please remove any secrets

Kirk Stennett · Answer

That s all it does

johann · Answer

Gotcha would you mind sharing your values yaml again please watch out for any secrets

Kirk Stennett · Answer

sure

johann · Answer

Ah I see the command I was thinking of was `helm template` which shouldn t error

Kirk Stennett · Answer

Do you want that full template yaml output

johann · Answer

The values yaml is probably most useful but the output could help

Kirk Stennett · Answer

It should just be the default with the `startupProbe` sections removed

johann · Answer

Looking in to it I m seeing that the dagit and dagster daemon startupProbes are still enabled with these values

johann · Answer

Likely an issue here with our recommendation to just comment out that lines since those will fall back to the default Sorry you ve bumped in to this

johann · Answer

gt I got it working by pulling it locally and removing that block < Kirk Stennett> to be clear you only had to remove the probe on userDeployments When I `helm template` your values I see probes on dagit and the dagster daemon deployments

Kirk Stennett · Answer

Yeah I just removed that block from the deployment user yaml file and it works when I run it all locally with `helm install dagster f values yaml`

Kirk Stennett · Answer

And no worries I m new to this and just wanted to make sure I was going about it the right way

johann · Answer

Hmm this is strange Could you run `helm template dagster dagster dagster f ~ personal config test values yaml` and see if you reproduce my result of having 2 startupProbes

Kirk Stennett · Answer

It does I just looked through that after you mentioned it

Kirk Stennett · Answer

Oh huh I added the segment back in and it worked locally too the deployment user yaml segment I removed previously

johann · Answer

Did you possibly remove the blocks from `deployment dagit yaml` and `deployment daemon yaml` Otherwise it would seem like your install should have failed

Kirk Stennett · Answer

Now I m confused as to how that worked

Kirk Stennett · Answer

It didn t the first time I ran through and those blocks still have the ``` if Values dagit startupProbe startupProbe toYaml Values dagit startupProbe | nindent 12 end ```

Kirk Stennett · Answer

Is there a different version of the code on github vs helm Looks like there was a helm update for dagster but even with that it still throws the same error

johann · Answer

Hi Kirk not sure how this worked the first time The chart on helm is from the latest dagster release in this case last friday and I don t see any recent changes This is a bug on our side it seems disabling the startupProbes isn t possible in user values yaml currently I ve opened a tracking issue and I ll look to have a fix out tommorow <https github com dagster io dagster issues 3796> In the meantime if you re able to manually remove the startupProbe blocks from `deployment dagit yaml` and `deployment daemon yaml` I think that should get you started Thanks for your help debugging this

Kirk Stennett · Answer

Thanks Johann I appreciate your help with this