https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Pieter Custers

06/15/2022, 3:08 PM
Hi, we are experiencing a problem with the postgres volume filling up very quickly, not related to data but related to logs. The root cause seems that Dagster is connecting many times per second and postgres logging all the (re)connections. Example log:
Copy code
2022-06-13 23:59:59.727 UTC,"dagster","dagster",2251490,"10.42.80.16:59264",62a7cf7f.225ae2,3,"idle",2022-06-13 23:59:59 UTC,,0,LOG,00000,"disconnection: session time: 0:00:00.011 user=dagster database=dagster host=10.42.80.16 port=59264",,,,,,,,,""
It is possible to disable this logging in Postgres, but not recommended. We also tried configuring this through dagster but we didn’t find any options. Do you have any best practices to prevent this from happening?
Note this part
session time: 0:00:00.011
. There seems to be many super short sessions… Why does the connection not persist?
p

prha

06/15/2022, 4:16 PM
Hi Pieter. What version of postgres and what version of Dagster are you using?
p

Pieter Custers

06/16/2022, 6:47 AM
Dagster 0.14.20, postgres 14
@prha did you have a chance to look at it
p

prha

06/17/2022, 7:14 PM
Hi Pieter. It looks like we are using connection pooling for some class of queries, but not in all places (e.g. we’re creating new connections when you view a run, for example)…. I can try to track down a solution, but I’ll create an issue for now since it might not be a quick fix.
@Dagster Bot issue Not using pooled PG connections in some Dagit queries
d

Dagster Bot

06/17/2022, 7:15 PM
p

Pieter Custers

06/20/2022, 7:02 AM
Thanks a lot!
b

Bernardo Cortez

10/26/2022, 5:25 PM
Hi @prha any news on this? We are facing similar problems
p

prha

10/27/2022, 1:06 AM
No new updates… but I’ll try to spend some more hours on this issue this week
👏 1
7 Views