I had two ops in a job
Flow of my job is execute op1 then op2
My op1 is performing melatno EL which is executing huge amount of data.
I had one question.
Suppose my job failed in middle stage of op1 ie melatno EL .
Now when I will execute my job again using option Rexecute 1 run from failure/ what it will do? Will it start op1 from beginning or from the state where op1 has failed.
If it will start from beginning then how can make it to execute to start it from the failed state.
Please help me out.
12/01/2022, 2:28 AM
The dagster machinery will start op 1 from the beginning. If you want to start from the failed state, you need to encode that in the body of the op itself by storing some sort of cursor between runs that's able to pick up where the failure left off. Where and how you do that depends on your use case, and your execution environment.