https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alexander Fife

09/15/2022, 7:46 AM
Hi everyone 🙂 I've got a partitioned SDA that ingests daily data from an API (full refreshes due to how the API is designed). However, several assets that rely on this data only needs the last/newest partition. It seems like assets downstream from a partitioned asset needs to be partitioned as well, however this just adds extra complexity and unnecessary load on our infrastructure on downstream assets. I see a couple of options and would love to hear what you guys think: 1. Is there a way to just specify that this asset just requires the last instantiation of an asset (or todays instantiation)? Then all further downstream assets can stay unpartitioned as well. 2. Just ignore it and do full refreshes of the io. Is this a common solution? 3. Add some kind of "Is this a backfill?" check to my backfills and return when it's a backfill. This solves uneccessary io but seems a bit fragile and still adds the complexity of dealing with partitions on downstream assets. 4. Add this to some kind of persistent storage and create a new asset that fetches it with a shared key. However, this creates a dependency that is not modeled or caught by Dagster. Does anyone have any tips to this type of ETL pattern in Dagster? Cheers,
2 Views