0 8 7 with postgres for run event log and schedule storage G

Question

0 8 7 with postgres for run event log and schedule storage Getting a too many clients postgres exception from sqlalchemy as an Engine Event in the pipeline run log Pgadmin shows that most of the ~100 idle connections to the `dagster` db are running the query `LISTEN run events ` I think 100 is the max clients setting for the postgres docker image I m running so it might be opening as many as it can get Is that normal

Danny · Answer

Sorry posted the wrong part of the trace Here s the full one

Danny · Answer

```dagster 1 | ERROR root psycopg2 OperationalError FATAL sorry too many clients already dagster 1 | dagster 1 | Background on this error at <http sqlalche me e 13 e3q8> dagster 1 | Traceback most recent call last dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine base py line 2339 in wrap pool connect dagster 1 | return fn dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 364 in connect dagster 1 | return ConnectionFairy checkout self dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 778 in checkout dagster 1 | fairy = ConnectionRecord checkout pool dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 495 in checkout dagster 1 | rec = pool do get dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool impl py line 239 in do get dagster 1 | return self create connection dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 309 in create connection dagster 1 | return ConnectionRecord self dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 440 in init dagster 1 | self connect first connect check=True dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 661 in connect dagster 1 | pool logger debug Error on connect s e dagster 1 | File usr local lib python3 7 site packages sqlalchemy util langhelpers py line 69 in exit dagster 1 | exc value with traceback=exc tb dagster 1 | File usr local lib python3 7 site packages sqlalchemy util compat py line 178 in raise dagster 1 | raise exception dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 656 in connect dagster 1 | connection = pool invoke creator self dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine strategies py line 114 in connect dagster 1 | return dialect connect cargs cparams dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine default py line 493 in connect dagster 1 | return self dbapi connect cargs cparams dagster 1 | File usr local lib python3 7 site packages psycopg2 init py line 127 in connect dagster 1 | conn = connect dsn connection factory=connection factory kwasync dagster 1 | psycopg2 OperationalError FATAL sorry too many clients already dagster 1 | dagster 1 | dagster 1 | The above exception was the direct cause of the following exception dagster 1 | dagster 1 | Traceback most recent call last dagster 1 | File usr local lib python3 7 site packages dagster celery core execution loop py line 108 in core celery execution loop dagster 1 | step key=step key dagster 1 | File usr local lib python3 7 site packages dagster core events init py line 644 in engine event dagster 1 | step key=step key dagster 1 | File usr local lib python3 7 site packages dagster core events init py line 199 in from pipeline dagster 1 | log pipeline event pipeline context event step key dagster 1 | File usr local lib python3 7 site packages dagster core events init py line 138 in log pipeline event dagster 1 | step key=step key dagster 1 | File usr local lib python3 7 site packages dagster core log manager py line 262 in debug dagster 1 | return self log logging DEBUG msg kwargs dagster 1 | File usr local lib python3 7 site packages dagster core log manager py line 232 in log dagster 1 | logger log level message extra=extra dagster 1 | File usr local lib python3 7 logging init py line 1444 in log dagster 1 | self log level msg args kwargs dagster 1 | File usr local lib python3 7 logging init py line 1514 in log dagster 1 | self handle record dagster 1 | File usr local lib python3 7 logging init py line 1524 in handle dagster 1 | self callHandlers record dagster 1 | File usr local lib python3 7 logging init py line 1586 in callHandlers dagster 1 | hdlr handle record dagster 1 | File usr local lib python3 7 logging init py line 894 in handle dagster 1 | self emit record dagster 1 | File usr local lib python3 7 site packages dagster core instance init py line 120 in emit dagster 1 | self instance handle new event event dagster 1 | File usr local lib python3 7 site packages dagster core instance init py line 868 in handle new event dagster 1 | self event storage store event event dagster 1 | File usr local lib python3 7 site packages dagster postgres event log event log py line 92 in store event dagster 1 | sql statement returning SqlEventLogStorageTable c run id SqlEventLogStorageTable c id dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine base py line 2237 in execute dagster 1 | connection = self contextual connect close with result=True dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine base py line 2305 in contextual connect dagster 1 | self wrap pool connect self pool connect None dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine base py line 2343 in wrap pool connect dagster 1 | e dialect self dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine base py line 1585 in handle dbapi exception noconnection dagster 1 | sqlalchemy exception with traceback=exc info 2 from =e dagster 1 | File usr local lib python3 7 site packages sqlalchemy util compat py line 178 in raise dagster 1 | raise exception dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine base py line 2339 in wrap pool connect dagster 1 | return fn dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 364 in connect dagster 1 | return ConnectionFairy checkout self dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 778 in checkout dagster 1 | fairy = ConnectionRecord checkout pool dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 495 in checkout dagster 1 | rec = pool do get dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool impl py line 239 in do get dagster 1 | return self create connection dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 309 in create connection dagster 1 | return ConnectionRecord self dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 440 in init dagster 1 | self connect first connect check=True dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 661 in connect dagster 1 | pool logger debug Error on connect s e dagster 1 | File usr local lib python3 7 site packages sqlalchemy util langhelpers py line 69 in exit dagster 1 | exc value with traceback=exc tb dagster 1 | File usr local lib python3 7 site packages sqlalchemy util compat py line 178 in raise dagster 1 | raise exception dagster 1 | File usr local lib python3 7 site packages sqlalchemy pool base py line 656 in connect dagster 1 | connection = pool invoke creator self dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine strategies py line 114 in connect dagster 1 | return dialect connect cargs cparams dagster 1 | File usr local lib python3 7 site packages sqlalchemy engine default py line 493 in connect dagster 1 | return self dbapi connect cargs cparams dagster 1 | File usr local lib python3 7 site packages psycopg2 init py line 127 in connect dagster 1 | conn = connect dsn connection factory=connection factory kwasync ```

Danny · Answer

Doesn t seem to be a problem on 0 8 5

Danny · Answer

On 0 8 7 I m seeing the connection count grow continuously as my pipeline progresses until it hits that 100 limit and the dagit engine starts erroring out It s not preallocating connections or anything it looks like a leak

Danny · Answer

On 0 8 5 with the same pipeline running pg connection counts to the dagster db typically stay at lt 10

Danny · Answer

Possibly relevant this pipeline continuously polls dagster s run list with the `wait for run slot` function here <https github com dagster io dagster issues 2716 issue 656821567> Don t imagine a lot of setups do this so thought I d mention this If there s any reason why this would stop working gt 0 8 5 that would probably be the answer

max · Answer

< prha> does this ring any bells

prha · Answer

no not off the top off my head

max · Answer

< Danny> i wonder about that call to `DagsterInstance get`

max · Answer

what if you passed the instance object through from the solid compute context instead

max · Answer

`context instance`

Danny · Answer

Yep will try it later and post back But I have the same thing implemented via graphql as well pipelineRunsOrError so I hope it doesn t also affect that Won t have context instance there Might be worth looking into why there s a difference 0 8 5 gt 0 8 7 seems like this might bite elsewhere

max · Answer

in the solid logic itself i mean

max · Answer

so this was new in 0 8 7

Danny · Answer

My solid is polling a dagster instance If it s a local instance it uses the implementation in the ticket I linked If remote it uses a sister implementation that uses graphql So I ll definitely try the context instance tip but it can only work in the former case

Danny · Answer

Yes new in 0 8 6 or 7

Danny · Answer

I can confirm issue isn t present 0 8 5

Danny · Answer

< prha> < max> I ve isolated the issue and its 100 reproducible in this repo The connection leak wasn t present in 0 8 6 or earlier only started in 0 8 7 Current 8 release still has the problem

Danny · Answer

I put repro instructions in the readme everything runs in docker so all you need is docker compose

Danny · Answer

Would be awesome if someone can take a look at this We need access to the excellent multidep fan in behavior in the latest releases so we can t stay on 0 8 6 So this connection leak is a blocker for us

Danny · Answer

TIA

Danny · Answer

Sweet I ve been able to track this down to the problematic commit This is the first commit that shows the db connection leak Binary search trial and error ftw So now the question is what in this commit can cause the symptoms I describe above steadily growing postgres connection count to dagster s database until it maxes out at 100 and Pgadmin shows that most of the ~100 idle connections are running the query `LISTEN run events ` Unfortunately I don t have the knowledge of dagster internals to tease this out any help would be appreciated I ll soon update the repo I used to track this down so that you guys can easily repro locally using this bad commit versus the one immediately before

max · Answer

yikes

max · Answer

< schrockn> < alex>

daniel · Answer

Opening an issue for this now and will investigate right away Thanks so much for finding the blamerev Danny

Danny · Answer

My pleasure thanks for looking into this partydagster

daniel · Answer