Alvaro Gomez Covella03/16/2023, 6:59 PM
that runs at daily basis that every day re-execute/materialized the last N days (fix number). We already have a Dagster pipeline that used assets (
) up and running. We even can achieve this with
. But for whatever reason, I’m not able to create a job (
) that backfill N days.
I’ve found this but I don’t know if it is exactly what I want. Means re-write our custom IOManager. Also found this unresolved open issue that might help. But I cannot find anything related to
in the docs.
I’m coming from Airflow world, so the first idea that come to my mind is to have a daily normal
that handles yesterday, and a second
that “clears” the last week execution. Is this possible in Dagster? or better, What is the most elegant way in Dagster?
yuhan03/16/2023, 9:21 PM
Alvaro Gomez Covella03/17/2023, 9:46 AM
Alexander Kerckhofs03/17/2023, 10:46 AM
Alvaro Gomez Covella03/17/2023, 10:47 AM