https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chad LeJeune

05/03/2023, 2:50 AM
Hello all! Is there a way to map an upstream asset to the latest day’s partition of a downstream asset? I am pulling data from an api that I can extract only the current state of all the records. I need to keep track of deleted records, so I was hoping to create file partitions for each day I pulled each endpoint. Then, I’d create a downstream asset that can be materialized based on the difference between the two upstream assets. I think the second part can be accomplished using partition mapping with date offsets, however, I can’t figure out how to create the initial daily file partition (I’d like it to work downstream with UPathIOManagers). In the documentation, I read that there are partition mapping definitions that can map the whole downstream asset to an upstream asset’s latest partition, or all partitions, but I would need the opposite: the latest partition of a downstream asset mapped a whole upstream asset (representing current state).
3 Views