I have dagit deployed on a Kubernetes cluster and ...
# announcements
b
I have dagit deployed on a Kubernetes cluster and it's regularly being restarted due to failure to respond to health checks (liveness probes), but there's no logs to indicate what's going on inside dagit's web server. is there any way to enable those logs?
a
from what i can tell it looks like we take
livenessProbe
as a helm value and don’t provide one - what check are you using?
b
it's just a http get on /
a
hmm - whats the timeout?
b
the check is generally OK until a few people try to browse to the UI, at which point it starts to fail
a
I recommend switching to
/dagit_info
until a few people try to browse to the UI
alright ya we have some known problematic gevent configuration issues, which this sounds like
b
ah that looks much better than using
/
a
so I think you’ll need a generous timeout on the GET
b
i'll up the timeout too, think it's at the default which is like 1s 😕
cheers alex
a
for our accidental-serial-web-server
😕
b
🙈
as an aside though, is there any way to turn on the webserver logs?
a
i don’t think so - we should wire something up
assuming flask has something we just need to enable