19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hello again! I'm running dagster with a databricks...
# ask-communityb
Bernardo Caldas
03/31/2022, 12:14 PMHello again! I'm running dagster with a databricks step launcher. The jobs are launched and ran in the daemons. Another issue I'm having is that sometimes my daemon dies but dagit seems to think the job is still running. How can I make it detect thar the job has terminated and that it has to run it again?
j
johann
03/31/2022, 4:49 PMHi Bernardo, how do you have Dagster deployed? What version are you on?
johann
03/31/2022, 4:52 PMWhat does it mean for your daemon to die? The box it’s running on goes down or the Dagster Daemon process crashes?
b
Bernardo Caldas
04/01/2022, 10:56 AMHi! Im on 0.14.6. It's running on k8s and sometimes it gets OOMkilled, so the whole box dies. Then jt is restarted, but dagit thinks the job is still running.
j
johann
04/01/2022, 4:41 PMAnd what run launcher are you using? I’m asking because with the helm chart it will default to the K8sRunLauncher which spins up runs in new K8s Jobs, so an OOM in the daemon shouldn’t impact them
johann
04/01/2022, 4:42 PMIf using the DefaultRunLauncher, the runs will be in subprocesses of the daemon and I can imagine that causing an OOM kill
johann
04/01/2022, 4:42 PMWe do have some new features in this area- note that they’re mostly oriented around the K8sRunLauncher https://docs.dagster.io/deployment/run-monitoring
b
Bernardo Caldas
04/05/2022, 5:27 PMIt's the defaultrunlauncher
Bernardo Caldas
04/05/2022, 5:28 PMI'm just worried that a daemon may die and it isn't detected
j
johann
04/05/2022, 5:36 PMGotcha, we don’t currently have run monitoring support on the DefaultRunLauncher. In a lot of cases the Daemon will receive an interrupt and it will fail the runs gracefully, but this will be an issue for an ungraceful shutdown
b
Bernardo Caldas
04/06/2022, 2:33 PMGot it... Anything you suggest that would make this a bit safer?
j
johann
04/06/2022, 2:37 PMUltimately we just need to add monitoring for the DefaultRunLauncher. In the meantime it would be possible to craft run timeouts etc. inside a scheduled Job or something like that
2 Views