https://dagster.io/ logo
Title
a

Alec Koumjian

02/08/2023, 8:37 PM
Quickly checking, is there anything that supports retaining a history of asset materializations? Our use case: we have an external source we are querying that we want to keep entire snapshots of over time.
:dagster-bot-resolve: 1
:dagster-bot-responded-by-community: 1
If we implement this ourselves, are there recommendations? Dynamic
AssetKey
with the date?
a

Adam Bloom

02/08/2023, 9:37 PM
A partitioned asset may be what you’re looking for. https://docs.dagster.io/concepts/partitions-schedules-sensors/partitions If the snapshots occur on a regular time interval, those could be your partitions
a

Alec Koumjian

02/08/2023, 10:08 PM
We are already using partitioned assets but not sure it’s appropriate in this case. The upstream resource changes day to day and we want to snapshot moments in time. If we set it up as partitions, it would break if we ever reran a partition by accident (since it will be replaced with the current days info)
a

Adam Bloom

02/08/2023, 10:09 PM
you could add a check at the start of your asset/op that the partition key is valid to prevent that
a

Alec Koumjian

02/08/2023, 10:10 PM
So basically if partition key != today then NOOP
s

sandy

02/10/2023, 4:58 PM
I don't have a better answer than @Adam Bloom, but I filed an issue to track supporting this in a more first-class way: https://github.com/dagster-io/dagster/issues/12253
a

Alec Koumjian

02/10/2023, 5:59 PM
Thank you 👍🏼