Is there a way to persist partial successes across re-runs? Concretely, I have an object I want to register in the database. It has a unique name, so if I try to re-run the exact same run, that registration process will fail. I’d like to be able to re-use the registered object across re-runs (e.g. if a later step fails), but still force people to update the name across different runs. Currently I’m forcing every run of that op to have a unique name, but I’d like to be able to do some sort of parent lineage tracing and can’t seem to find any useful helper functions that I know must exist.

When re-executing from failure, dagster will only re-execute steps that failed, so if the op that registers the object completed successfully, re-executions should just be using that object.

I don’t think I articulated my question well. There’s not an issue with the re-execution from failure logic. I want to be able to execute

All Steps in Root Run

or

Same Steps

without editing the launchpad and be able to use the successful outcome from before OR I want to be able to execute

From Failure

but be able to edit the launchpad configuration to update the name of the object

Gotcha - how are you setting the name right now? Via config? Can you not just pick the subset you want to execute, and run with the same config? You could check to see whether the database object already exists, and if so ignore it

Yes, currently it’s set via the op’s config. The issue with checking if something already exists is that I want to do that check depending on the type of action someone took from dagster. Sometimes I want it to fail (e.g. if it’s a new run and the object already exists), but sometimes I want to do what you suggest (check for existence and then use existing one) — it depends on the way the pipeline was triggered.

ahh I see. We definitely don't have any out of the box functionality to determine whether your run came from "re-execution" vs just using the launchpad. You could possibly infer it from munging around with the internals, but I'm hesitant to recommend that as those are subject to change without warning. One thing you could do here is set a tag when running from particular places, which you could use in the body of your op to determine where you are running from.

Is there a way to re-execute from Failure with a new config, but not re-execute the already successful steps?