Cameron Gallivan
09/15/2021, 12:22 AMWARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
2021-09-14 07:20:20 - SensorDaemon - ERROR - Sensor daemon caught an error for sensor dragen_postprocessing_sort_sqs_sensor : dagster_postgres.utils.DagsterPostgresException: too many retries for DB connection
Stack Trace:
File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 230, in execute_sensor_iteration
yield from _evaluate_sensor(
ERROR:SensorDaemon:Sensor daemon caught an error for sensor dragen_postprocessing_sort_sqs_sensor : dagster_postgres.utils.DagsterPostgresException: too many retries for DB connection
Stack Trace:
File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 230, in execute_sensor_iteration
yield from _evaluate_sensor(
File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 114, in __exit__
self._write()
File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 85, in _write
self._instance.update_job_tick(self._tick)
File "/opt/venv/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 1529, in update_job_tick
return self._schedule_storage.update_job_tick(tick)
File "/opt/venv/lib/python3.8/site-packages/dagster/core/storage/schedules/sql_schedule_storage.py", line 196, in update_job_tick
with self.connect() as conn:
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 160, in create_pg_connection
conn = retry_pg_connection_fn(engine.connect)
File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 127, in retry_pg_connection_fn
raise DagsterPostgresException("too many retries for DB connection") from exc
The above exception was the direct cause of the following exception:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "docker_postgresql" to address: Temporary failure in name resolution
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)
Stack Trace:
File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 116, in retry_pg_connection_fn
return fn()
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
return self._connection_cls(self, close_with_result=close_with_result)
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
else engine.raw_connection()
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
return self._wrap_pool_connect(self.pool.connect, _connection)
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3215, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2069, in _handle_dbapi_exception_noconnection
util.raise_(
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
return fn()
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 307, in connect
return _ConnectionFairy._checkout(self)
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
rec = pool._do_get()
File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
return self._create_connection()
File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 114, in __exit__
self._write()
File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 85, in _write
self._instance.update_job_tick(self._tick)
File "/opt/venv/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 1529, in update_job_tick
return self._schedule_storage.update_job_tick(tick)
File "/opt/venv/lib/python3.8/site-packages/dagster/core/storage/schedules/sql_schedule_storage.py", line 196, in update_job_tick
with self.connect() as conn:
File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 160, in create_pg_connection
conn = retry_pg_connection_fn(engine.connect)
File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 127, in retry_pg_connection_fn
raise DagsterPostgresException("too many retries for DB connection") from exc