dagster.daemon - ERROR - Stopping dagster-daemon p...
# ask-community
n
dagster.daemon - ERROR - Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['SENSOR', 'SCHEDULER'] EDIT-- I checked for more recent versions of dagster and there is a bugfix for this in a newer version 1.1.13 - I will try the newest version and report back in this thread. -- My dagster-daemon is dying fairly regularly now and it's distrupting my day-to-day operations, we end up losing ~8 hours of overnight processing when this happens. I need to figure out why dagster-daemon is dying and address the root cause. Please help! Thanks, Nick
Copy code
dagster.daemon - ERROR - Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['SENSOR', 'SCHEDULER']
dagster.daemon - INFO - Shutting down daemon threads...
dagster.daemon - INFO = Daemon threads shut down.
🤖 1
s
Hi Nicholas, Good that you’ve identified an update may fix the issue-- did you try it yet?
👍 1
n
I did apply the update, but because this issue was occurring a couple times a week, I have to wait and see if the issue is fixed. So far things are good but I am getting an unrelated error : unexpected graphql - it doesn't appear to be affecting the rest of dagster - my scheduler seems to be working fine.
Copy code
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: kvs
[SQL: SELECT kvs."key", kvs.value 
FROM kvs 
WHERE kvs."key" IN (?)]
[parameters: ('ASSET_DAEMON_PAUSED',)]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)
image.png
when i try to turn off auto-materializing i get the error above
s
hmmm, cc @johann as the auto-materialize daemon guru
j
The KVS table that you’re missing was introduced in dagster 0.15- you can run
dagster instance migrate
in the same environment that you run dagit or use https://docs.dagster.io/deployment/guides/kubernetes/how-to-migrate-your-instance if you’re on K8s. It’s never a bad idea to make a db backup first
n
got it , ty , will try it out after my in-process runs
s
Hey Nicholas, just checking if you’ve resolved the issue?
n
Hi sean, yes the errors related to
Copy code
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: kvs
[SQL: SELECT kvs."key", kvs.value 
FROM kvs 
WHERE kvs."key" IN (?)]
[parameters: ('ASSET_DAEMON_PAUSED',)]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)
Have been resolved ! 👍
🙌 1
It has been about a week now, I am feeling confident that the dagster update resolved the scheduler daemon issue I was having. Tyvm for your help!
151 Views