Hi, I'm running Dagster 1.3.5 on Kubernetes cluster.

dagster-daemon

continuously crashes with error

dagster.daemon - ERROR - Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['BACKFILL']
dagster.daemon - WARNING - Shutting down daemon threads due to Exception...
dagster.daemon - ERROR - Thread for BACKFILL did not shut down gracefully.
dagster.daemon - INFO - Daemon threads shut down.

Dagster is deployed using Helm chart (

heartbeatTolerance: 300

). The problem consistently occurs in our Prod environment (even if there are no active job executions), never occurs in Staging. The notable difference between Prod and Staging is the size of database (>10GB in Prod) and number of Backfills processed (>100 in Prod). Any ideas how to fix it?