https://dagster.io/ logo
#ask-community
Title
# ask-community
r

Roel Hogervorst

04/06/2022, 11:42 AM
I'm on dagster 0.14.5 and when a run is going on I have to manually refresh the dagit page to see the logs coming in. (this runs on prem on a kubernetes cluster) Is this expected behavior?
j

johann

04/06/2022, 12:40 PM
That isn't expected. Do you see any errors in your browser console or the Dagit server
r

Roel Hogervorst

04/06/2022, 12:44 PM
I see no errors in de browser
I do get sqlalchemy errors all the time,
j

johann

04/06/2022, 1:03 PM
Could you share an example
r

Roel Hogervorst

04/06/2022, 2:06 PM
(I'm not sure if the SQLalchemy errors are related but I'll type them over) I get errors like: OperationalError: server closed connection unexpectedly: [SQL: select jobs.job_body, jobs.job_origin_id form jobs where jobs.repository_origin_id <parameters here>]
There are never any errors in the database log (postgres on the same cluster)
I've increased the cpu and memory of the database statefulset
this cluster uses istio to encrypt all connections between pods
The dagster deamon never has any issues, the logs only say that there are no runs because there are no sensors
The usercode repository deployment has has no errors either just logs of engine events
j

johann

04/06/2022, 2:14 PM
Loosing a DB connection seems like a likely culprit for breaking the Dagit refreshes. Are you able to correlate those logs with the behavior at all? What sqlalchemy and postgresql versions are you on
r

Roel Hogervorst

04/06/2022, 3:10 PM
postgres version 14 I don't know the sqlalchemy version, whatever version is installed with dagit/ dagster 0.14.5
j

johann

04/08/2022, 9:21 PM
Is it possible your db is under heavy load that’s making the connections timeout?
2 Views