19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
I'm on dagster 0.14.5 and when a run is going on I...
# ask-communityr
Roel Hogervorst
04/06/2022, 11:42 AMI'm on dagster 0.14.5 and when a run is going on I have to manually refresh the dagit page to see the logs coming in. (this runs on prem on a kubernetes cluster) Is this expected behavior?
j
johann
04/06/2022, 12:40 PMThat isn't expected. Do you see any errors in your browser console or the Dagit server
r
Roel Hogervorst
04/06/2022, 12:44 PMI see no errors in de browser
Roel Hogervorst
04/06/2022, 12:46 PMI do get sqlalchemy errors all the time,
j
johann
04/06/2022, 1:03 PMCould you share an example
r
Roel Hogervorst
04/06/2022, 2:06 PM(I'm not sure if the SQLalchemy errors are related but I'll type them over) I get errors like: OperationalError: server closed connection unexpectedly: [SQL: select jobs.job_body, jobs.job_origin_id form jobs where jobs.repository_origin_id <parameters here>]
Roel Hogervorst
04/06/2022, 2:06 PMThere are never any errors in the database log (postgres on the same cluster)
Roel Hogervorst
04/06/2022, 2:06 PMI've increased the cpu and memory of the database statefulset
Roel Hogervorst
04/06/2022, 2:08 PMthis cluster uses istio to encrypt all connections between pods
Roel Hogervorst
04/06/2022, 2:08 PMThe dagster deamon never has any issues, the logs only say that there are no runs because there are no sensors
Roel Hogervorst
04/06/2022, 2:10 PMThe usercode repository deployment has has no errors either just logs of engine events
j
johann
04/06/2022, 2:14 PMLoosing a DB connection seems like a likely culprit for breaking the Dagit refreshes. Are you able to correlate those logs with the behavior at all? What sqlalchemy and postgresql versions are you on
r
Roel Hogervorst
04/06/2022, 3:10 PMpostgres version 14
I don't know the sqlalchemy version, whatever version is installed with dagit/ dagster 0.14.5
j
johann
04/08/2022, 9:21 PMIs it possible your db is under heavy load that’s making the connections timeout?
2 Views