Hi I am trying to run a pipeline and after some time not alw

Question

Hi I am trying to run a pipeline and after some time not always the same sometimes 1 30 minutes other times 10 mins I get a postgres connection error and the pipeline fails The pipeline consists of seven solids and each one of them executes a bunch of queries to BigQuery The whole pipeline on production should take a couple of hours to finish We don t have this problem with our other Dagster pipelines which are a lot smaller and shorter in time Any help is greatly appreciated thanks

Andreas Williams · Answer

here s the error stack trace

Andreas Williams · Answer

```File usr local lib python3 8 site packages dagster core execution plan execute plan py line 190 in dagster event sequence for step for step event in check generator step events File usr local lib python3 8 site packages dagster core execution plan execute step py line 312 in core dagster event sequence for step for user event in check generator File usr local lib python3 8 site packages dagster core execution plan execute step py line 71 in step output error checked user event sequence for user event in user event sequence File usr local lib python3 8 site packages dagster core execution plan execute step py line 599 in user event sequence for step compute fn for event in iterate with context File usr local lib python3 8 site packages dagster utils init py line 362 in iterate with context next output = next iterator File usr local lib python3 8 site packages dagster core execution plan compute py line 126 in execute core compute for step output in yield compute results compute context inputs compute fn File usr local lib python3 8 site packages dagster core execution plan compute py line 92 in yield compute results user event generator = compute fn SolidExecutionContext compute context inputs File usr local lib python3 8 site packages dagster core definitions decorators solid py line 304 in compute result = fn context kwargs File customer platform dagster pipelines pipeline scv mvp oneoff py line 169 in hygiene 01 execute context File customer platform dagster pipelines scv mvp oneoff poc utils py line 21 in decorator return func args kwargs File customer platform dagster pipelines scv mvp oneoff poc run poc py line 111 in execute utils assert equal context File customer platform dagster pipelines scv mvp oneoff poc utils py line 117 in assert equal result query details = execute sql context kwargs File customer platform dagster pipelines scv mvp oneoff poc utils py line 83 in execute sql File usr local lib python3 8 site packages dagster core log manager py line 243 in info return self log msg kwargs File usr local lib python3 8 site packages dagster core log manager py line 204 in log logger log level message extra=extra File usr local lib python3 8 logging init py line 1500 in log self log level msg args kwargs File usr local lib python3 8 logging init py line 1577 in log self handle record File usr local lib python3 8 logging init py line 1587 in handle self callHandlers record File usr local lib python3 8 logging init py line 1649 in callHandlers hdlr handle record File usr local lib python3 8 logging init py line 950 in handle self emit record File usr local lib python3 8 site packages dagster core instance init py line 152 in emit self instance handle new event event File usr local lib python3 8 site packages dagster core instance init py line 1087 in handle new event self event storage store event event File usr local lib python3 8 site packages dagster postgres event log event log py line 139 in store event with self connect as conn File usr local lib python3 8 contextlib py line 113 in enter return next self gen File usr local lib python3 8 site packages dagster postgres utils py line 157 in create pg connection conn = retry pg connection fn engine connect File usr local lib python3 8 site packages dagster postgres utils py line 113 in retry pg connection fn return fn File usr local lib python3 8 site packages sqlalchemy engine base py line 3095 in connect return self connection cls self close with result=close with result File usr local lib python3 8 site packages sqlalchemy engine base py line 91 in init else engine raw connection File usr local lib python3 8 site packages sqlalchemy engine base py line 3174 in raw connection return self wrap pool connect self pool connect connection File usr local lib python3 8 site packages sqlalchemy engine base py line 3141 in wrap pool connect return fn File usr local lib python3 8 site packages sqlalchemy pool base py line 301 in connect return ConnectionFairy checkout self File usr local lib python3 8 site packages sqlalchemy pool base py line 755 in checkout fairy = ConnectionRecord checkout pool File usr local lib python3 8 site packages sqlalchemy pool base py line 419 in checkout rec = pool do get File usr local lib python3 8 site packages sqlalchemy pool impl py line 259 in do get return self create connection File usr local lib python3 8 site packages sqlalchemy pool base py line 247 in create connection return ConnectionRecord self File usr local lib python3 8 site packages sqlalchemy pool base py line 362 in init self connect first connect check=True File usr local lib python3 8 site packages sqlalchemy pool base py line 599 in connect connection = pool invoke creator self File usr local lib python3 8 site packages sqlalchemy engine create py line 578 in connect return dialect connect cargs cparams File usr local lib python3 8 site packages sqlalchemy engine default py line 548 in connect return self dbapi connect cargs cparams File usr local lib python3 8 site packages psycopg2 init py line 127 in connect conn = connect dsn connection factory=connection factory kwasync File usr local lib python3 8 site packages dagster utils interrupts py line 78 in new signal handler raise error cls ```

Andreas Williams · Answer

cc < Sri Kadiyala>

daniel · Answer

Hi Andreas thanks for the stack trace do you have the text of the error message as well just above the trace Might help us narrow it down

Andreas Williams · Answer

yes sorry forgot to copy that too Here `dagster core errors DagsterExecutionInterruptedError`

daniel · Answer

Interesting so that error should only be triggered if your pipeline process receives an interrupt signal somehow which shouldn t be initiated from Dagster unless you attempted to manually terminate the pipeline Is there anything else you can think of that would be trying to kill or terminate the pipeline process

Andreas Williams · Answer

hmm interesting the only other thing I can think of is that there is an error with BigQuery sql execution or connection but whenever I had such errors in the past they were clearly reported in Dagit

alex · Answer

if there is nothing too sensitive in the event log you could send over a debug file There is a download debug file option in the ` ` menu on the runs page

Sri Kadiyala · Answer

Hello Alex here is the dbug file

alex · Answer

Thanks thats useful In this case kubernetes seems to be attempting to move the pod which we currently do not handle gracefully You may need to adjust any k8s autoscaling systems you are using or include tags on the dagster runs to prevent them participating