Alexis Manuel
01/19/2023, 7:51 AMTraceback (most recent call last):
File "/usr/local/bin/dagster-daemon", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/cli/__init__.py", line 127, in main
cli(obj={}) # pylint:disable=E1123
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/cli/__init__.py", line 43, in run_command
_daemon_run_command(instance, kwargs)
File "/usr/local/lib/python3.7/site-packages/dagster/_core/telemetry.py", line 110, in wrap
result = f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/cli/__init__.py", line 55, in _daemon_run_command
controller.check_daemon_loop()
File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/controller.py", line 268, in check_daemon_loop
self.check_daemon_heartbeats()
File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/controller.py", line 239, in check_daemon_heartbeats
raise Exception("Stopped dagster-daemon process due to thread heartbeat failure")
Exception: Stopped dagster-daemon process due to thread heartbeat failure
And:
Stopping dagster-daemon process since the following threads are no longer sending heartbeats: ['BACKFILL']
Shutting down daemon threads...
Thread for BACKFILL did not shut down gracefully.
For more context, it happened after I tried to launch a backfill of few runs.
How can I revert the daemon into a stable state ?daniel
01/19/2023, 3:32 PMAlexis Manuel
01/19/2023, 3:44 PMdaniel
01/19/2023, 3:46 PMAlexis Manuel
01/19/2023, 3:47 PMdaniel
01/19/2023, 4:15 PMAlexis Manuel
01/19/2023, 4:29 PMdaniel
01/19/2023, 4:40 PMAlexis Manuel
01/19/2023, 5:16 PMfifteen_minute_partitions = TimeWindowPartitionsDefinition(
cron_schedule="*/15 * * * *",
start=datetime(2022, 1, 1, 0, 0, 0),
fmt="%Y-%m-%d %H:%M",
timezone="Europe/Paris",
)
@asset(
group_name="group_name",
io_manager_key=settings.FS_IO_MANAGER,
key_prefix="some_prefix",
partitions_def=fifteen_minute_partitions,
required_resource_keys={"api"},
)
def tmp_asset(context) -> DataFrame:
"""Some desc."""
columns = [
"field1",
"field2"
]
start_dt, end_dt = context.output_asset_partitions_time_window()
req_params = RequestParameters(
...
)
return context.resources.api.fetch(context, req_params, columns)
sandy
01/19/2023, 5:16 PMAlexis Manuel
01/19/2023, 5:20 PMdaniel
01/19/2023, 5:22 PMBinoy Shah
01/19/2023, 5:47 PMStopped dagster-daemon process due to thread heartbeat failure
?daniel
01/19/2023, 5:50 PMBinoy Shah
01/19/2023, 5:52 PMdaniel
01/19/2023, 5:53 PMBinoy Shah
01/19/2023, 5:55 PMdaniel
01/19/2023, 5:55 PMBinoy Shah
01/19/2023, 5:58 PMdaniel
01/19/2023, 6:17 PMdagsterDaemon:
heartbeatTolerance:
Alexis Manuel
01/20/2023, 7:28 AM