Say if I have a daily pipeline that takes 10 hours...

# announcementsf

Felipe Saldana

02/05/2021, 10:02 PMSay if I have a daily pipeline that takes 10 hours to compute and one given task (solid A) takes 3 hours. But solid A's data only updates once a week. What mechanism does Dagster have to let the pipeline know to use solid A's generated data set for a week instead of regenerating the same data daily and thus wasting that 3 hours.

2 Views