The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Screen Shot 2022-05-18 at 7.31.06 PM.png

Screen Shot 2022-05-18 at 7.30.38 PM.png

My job failed but didn’t retry
<https://nautilus.dagster.cloud/prod/instance/runs/cff862c3-77eb-45aa-828e-92eb3054729b?logKey=live_pricing_op_highmem>

Other jobs seem to be working fine
<https://nautilus.dagster.cloud/prod/instance/runs/d773f6cc-527a-49ad-998b-712ed2fe1add?logKey=live_pricing_op>

Screen Shot 2022-05-18 at 7.35.17 PM.png

the same instance of the live_pricing_op_highmem job from earlier DID retry

<https://nautilus.dagster.cloud/prod/instance/runs/802ff731-5b47-4bba-87ae-632eba0bcd3c?logKey=live_pricing_op_highmem>

Hmm, is it possible your agent went down around that time?

could be possible. is there a way to gracefully recover in that case?

guess it might require more of an external monitoring solution — use graphql to force a job to restart if the run post-conditions aren’t fulfilled?

Since it seems like this was an ephemeral issue, I think we should expand the retry policy to retry this type of failure (not being able to connect with your user code when launching the retry run)