https://dagster.io/ logo
Title
c

Cameron Gallivan

09/15/2021, 12:22 AM
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
WARNING:root:Retrying failed database connection
2021-09-14 07:20:20 - SensorDaemon - ERROR - Sensor daemon caught an error for sensor dragen_postprocessing_sort_sqs_sensor : dagster_postgres.utils.DagsterPostgresException: too many retries for DB connection

Stack Trace:
  File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 230, in execute_sensor_iteration
    yield from _evaluate_sensor(
ERROR:SensorDaemon:Sensor daemon caught an error for sensor dragen_postprocessing_sort_sqs_sensor : dagster_postgres.utils.DagsterPostgresException: too many retries for DB connection

Stack Trace:
  File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 230, in execute_sensor_iteration
    yield from _evaluate_sensor(
  File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 114, in __exit__
    self._write()
  File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 85, in _write
    self._instance.update_job_tick(self._tick)
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 1529, in update_job_tick
    return self._schedule_storage.update_job_tick(tick)
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/storage/schedules/sql_schedule_storage.py", line 196, in update_job_tick
    with self.connect() as conn:
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 160, in create_pg_connection
    conn = retry_pg_connection_fn(engine.connect)
  File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 127, in retry_pg_connection_fn
    raise DagsterPostgresException("too many retries for DB connection") from exc

The above exception was the direct cause of the following exception:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "docker_postgresql" to address: Temporary failure in name resolution

(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

Stack Trace:
  File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 116, in retry_pg_connection_fn
    return fn()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3215, in _wrap_pool_connect
    Connection._handle_dbapi_exception_noconnection(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2069, in _handle_dbapi_exception_noconnection
    util.raise_(
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
    return fn()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 307, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 425, in checkout
    rec = pool._do_get()
  File "/opt/venv/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 256, in _do_get
    return self._create_connection()
  File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 114, in __exit__
    self._write()
  File "/opt/venv/lib/python3.8/site-packages/dagster/daemon/sensor.py", line 85, in _write
    self._instance.update_job_tick(self._tick)
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/instance/__init__.py", line 1529, in update_job_tick
    return self._schedule_storage.update_job_tick(tick)
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/storage/schedules/sql_schedule_storage.py", line 196, in update_job_tick
    with self.connect() as conn:
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 160, in create_pg_connection
    conn = retry_pg_connection_fn(engine.connect)
  File "/opt/venv/lib/python3.8/site-packages/dagster_postgres/utils.py", line 127, in retry_pg_connection_fn
    raise DagsterPostgresException("too many retries for DB connection") from exc