https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
b

Binoy Shah

05/11/2022, 1:31 PM
Hi, We are using our custom Kubernetes cluster to deploy. Our kube enforces certain rules and that includes livenessProbe, readinessProbe to be included. It also needs resources to be defined, I am having issues with couple of things
Copy code
{
	"msg": "Deployment/RELEASE-NAME-dagster-daemon: No probe defined in dagster container. Must define either a readinessProbe or livenessProbe"
},
{
	"msg": "Pod/RELEASE-NAME-dagit-test-connection: No probe defined in wget container. Must define either a readinessProbe or livenessProbe"
},
{
	"msg": "Pod/RELEASE-NAME-dagit-test-connection: CPU requests is not defined"
},
{
	"msg": "Pod/RELEASE-NAME-dagit-test-connection: Memory requests is not defined"
},
{
	"msg": "Pod/RELEASE-NAME-dagit-test-connection: CPU limits is not defined"
},
{
	"msg": "Pod/RELEASE-NAME-dagit-test-connection: Memory limits is not defined"
},
{
	"msg": "Pod/RELEASE-NAME-dagit-test-connection: Does not have a serviceAccountName defined"
}
So why is the dagit-test-connection needed, can I turn it off ? What command would serve as a readiness/livenessProbe for dagster-daemon
d

daniel

05/11/2022, 3:27 PM
Hi Binoy - dagit-test-connection isn't needed for the helm chart to operate, but I don't think we currently have a flag that removes it. Is that something you can remove on your side for now while we look into this, e.g. with a tool like kustomize?
b

Binoy Shah

05/11/2022, 3:29 PM
Unfortunately, it is all via helm with us.. that’s why I am packaging it within my empty helm chart via Helm dependencies. @daniel
what command can serve as a livenessProbe for dagster-daemon ?
d

daniel

05/11/2022, 3:29 PM
its possible we can just remove dagit-test-connection from the chart
i can send you the original liveness probe that we used, although we removed it because it was sometimes causing more problems than it solved
b

Binoy Shah

05/11/2022, 3:31 PM
Oh, that would be great, if you can share the liveness probe, If you can recall, what problems were you facing with livenessProbe ?
@daniel
d

daniel

05/11/2022, 3:45 PM
no need to tag me i get notifications when you comment 🙂 I remember one problem was that the probe that we used reached out to the dagster DB to decide if the daemon was running, which meant that transient DB issues could also shut down the daemon. The daemon kind of implements its own liveness probe internally, in that it will shut itself down if it starts hanging. Is giving it a simple liveness probe that always returns True an option, or would that break your organization's policies?
https://github.com/dagster-io/dagster/pull/6553 has a pretty good explanation, that is the PR that removed the liveness probes by default
It looks like the original liveness probe before that PR was "dagster-daemon liveness-check"
b

Binoy Shah

05/11/2022, 3:48 PM
sorry about that tagging (force of habit), and thank you for the link, and yes, i’ll take look at the livenessProbe and if that there’s internal
livenessProbe
already in dagster daemon, then I’ll probably just do a command with
ls
to return success and satisfy the rules
What are you recommendations on disabling the
dagit-test-connection
pod ?
d

daniel

05/11/2022, 4:03 PM
i think we can remove it for you
if you can wait a day
b

Binoy Shah

05/11/2022, 4:04 PM
Oh wonderful, yes I can wait, I am still working on building my
values.yaml
file for deployment
Hi Daniel, do you have any update for me to disable the dagit-test-connection pods ?
d

daniel

05/12/2022, 2:14 PM
yes, that's going out in the release today
b

Binoy Shah

05/12/2022, 2:14 PM
cool, ty
2 Views