dagster.daemon - ERROR - Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['SENSOR', 'SCHEDULER'] EDIT-- I checked for more recent versions of dagster and there is a bugfix for this in a newer version 1.1.13 - I will try the newest version and report back in this thread. -- My dagster-daemon is dying fairly regularly now and it's distrupting my day-to-day operations, we end up losing ~8 hours of overnight processing when this happens. I need to figure out why dagster-daemon is dying and address the root cause. Please help! Thanks, Nick

dagster.daemon - ERROR - Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['SENSOR', 'SCHEDULER']
dagster.daemon - INFO - Shutting down daemon threads...
dagster.daemon - INFO = Daemon threads shut down.

Hi Nicholas, Good that you’ve identified an update may fix the issue-- did you try it yet?

I did apply the update, but because this issue was occurring a couple times a week, I have to wait and see if the issue is fixed. So far things are good but I am getting an unrelated error : unexpected graphql - it doesn't appear to be affecting the rest of dagster - my scheduler seems to be working fine.

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: kvs
[SQL: SELECT kvs."key", kvs.value 
FROM kvs 
WHERE kvs."key" IN (?)]
[parameters: ('ASSET_DAEMON_PAUSED',)]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

hmmm, cc @johann as the auto-materialize daemon guru

The KVS table that you’re missing was introduced in dagster 0.15- you can run

dagster instance migrate

in the same environment that you run dagit or use https://docs.dagster.io/deployment/guides/kubernetes/how-to-migrate-your-instance if you’re on K8s. It’s never a bad idea to make a db backup first

got it , ty , will try it out after my in-process runs

Hey Nicholas, just checking if you’ve resolved the issue?

Hi sean, yes the errors related to

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: kvs
[SQL: SELECT kvs."key", kvs.value 
FROM kvs 
WHERE kvs."key" IN (?)]
[parameters: ('ASSET_DAEMON_PAUSED',)]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

Have been resolved ! 👍