We are using the dagster helm chart in our own k8s...
# deployment-kubernetes
j
We are using the dagster helm chart in our own k8s cluster & cloud sql database. I just realised that since 2nd of May (when we upgraded from dagster version 1.1.20 to 1.3.1) the daemon pod is showing these gnarly errors
Copy code
2023-05-30 16:50:58 +0000 - dagster.daemon.EventLogConsumerDaemon - ERROR - Caught error:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "kvs" does not exist
LINE 2: FROM kvs
             ^

[SQL: SELECT kvs.key, kvs.value
FROM kvs
WHERE kvs.key IN (%(key_1_1)s, %(key_1_2)s)]
[parameters: {'key_1_1': 'EVENT_LOG_CONSUMER_CURSOR-PIPELINE_FAILURE', 'key_1_2': 'EVENT_LOG_CONSUMER_CURSOR-PIPELINE_SUCCESS'}]
(Background on this error at: <https://sqlalche.me/e/14/f405>)

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/daemon.py", line 222, in core_loop
    yield from self.run_iteration(workspace_process_context)
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/auto_run_reexecution/event_log_consumer.py", line 45, in run_iteration
    persisted_cursors = _fetch_persisted_cursors(instance, DAGSTER_EVENT_TYPES, self._logger)
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/auto_run_reexecution/event_log_consumer.py", line 111, in _fetch_persisted_cursors
    {_create_cursor_key(event_type) for event_type in event_types}
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/storage/runs/sql_run_storage.py", line 1146, in get_cursor_values
    KeyValueStoreTable.c.key.in_(keys)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 335, in _execute_on_connection
    self, multiparams, params, execution_options
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1582, in _execute_clauseelement
    cache_hit=cache_hit,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)

The above exception was caused by the following exception:
psycopg2.errors.UndefinedTable: relation "kvs" does not exist
LINE 2: FROM kvs
             ^


Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
my best guess is that the DB schema has changed. Question: is this guess correct? and is there a way to fix it by running some kind of DB migration?
nevermind, I think I found what I need: https://docs.dagster.io/migration#database-migration
❤️ 1