I don't know how plausible it would be but it'd be...
# dagster-feedback
z
I don't know how plausible it would be but it'd be really nice if Dagster could restart from failure when there is conditional branching in a workflow. we have a workflow that goes like maybe_run_on_platform_1 \ start < -> postprocessing to combine results maybe_run_on_platform_2 / and if only maybe_run_on_platform_1 executes but postprocessing fails, then I can't restart from failure because dagster can't find the state for maybe_run_on_platform_2
1
🤖 1
it seems like maybe dagster could hold a little more state about previous executions and determine that if a branch was skipped in the previous execution, it doesn't need to try to get the results from that branch. but I don't know the internals that well, maybe that kind of state isn't really maintained
m
I usually can't restart from failure (though I can re-run individual nodes) because Dagster can't find state. We have lots of steps that are sometimes skipped so I assume it's the same underlying constraint.
a
i tried to reproduce this and my test case is passing https://github.com/dagster-io/dagster/pull/11581/files are there more details
ty thankyou 1
m
I have a run that failed recently. Here's the error I get when I try to re-execute from failure:
Copy code
dagster._core.errors.DagsterInvariantViolationError: Unresolved ExecutionStep "data_pipe_graph_<zyx>.sanitation_graph._sanitize_task[?]" is resolved by "data_pipe_graph_<xyz>.sanitation_graph.yield_each_test" which is not part of the current step selection
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/impl.py", line 416, in get_external_execution_plan_snapshot
    create_execution_plan(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 1058, in create_execution_plan
    return ExecutionPlan.build(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/plan.py", line 1061, in build
    return plan_builder.build()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/plan.py", line 258, in build
    plan = plan.build_subset_plan(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/plan.py", line 842, in build_subset_plan
    executable_map, resolvable_map = _compute_step_maps(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/plan/plan.py", line 1505, in _compute_step_maps
    raise DagsterInvariantViolationError(
Feel free to check out https://formenergy.dagster.cloud/td-staging/runs/7b17259c-fc14-4b94-bc5a-3f8d00026d88?log[…]_reactor.aggregation_graph._aggregate_task%5BQLZ6J%5D%22 for more of that graph structure, but I have a couple dynamic fan outs. The
yield_each_test
it's complaining about yields dynamic outs for the sanitation stage, all the `_sanitize_task`s succeeded, then there's another fan-out for `_aggregation_task`s one of which is the one that failed on this run and is what I'd expect to be re-run with a "from failure".
a
ah dynamic is in the mix - i just landed some fixes around skip behavior that may impact this
👌 1
kept pulling this thread and have some fixes coming together https://github.com/dagster-io/dagster/pull/11581
ty thankyou 2