i'm planning on launching several jobs using hourly partitions. one thing i noticed is that if you attempt to re-fill a partition, all of the job nodes aren't cleared from the first run.
1. run partitioned job. everything compeltes successfully (all green)
2. re-run partitioned job, it fails. (one node red, rest are green)
i believe in airflow, re-triggering a partition is basically the same as marking all of those tagged runs as "cleared". not sure if there's a simple way to do this in dagster, but it makes for a nicer/more comprehensible visual in my opinion.
05/16/2022, 6:49 PM
05/16/2022, 7:32 PM
Hi Stephen. Just want to confirm what is confusing here: Some of the steps are retaining their “success” status from previous runs, when really you’d want to see the entire column just contain the status from the last run?
05/17/2022, 12:45 AM
that's right. my intuition is that at any given time, a partition should only have state from a single run (the latest). i don't expect this is a common problem in practice, because people wont be randomly kicking off a bunch of new backfills most of the time, but just wanted to share
05/17/2022, 3:42 PM
Yeah, that matches my intuition too… It’s a bug that it smushes them together.
@Dagster Bot issue Partition step matrix combines the per-step status across multiple runs