https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alessandro Cantarelli

09/15/2022, 2:50 PM
Hi all, is there a similar inbuild dagster function much like DailyPartitionsDefinition that however takes in a start and end date and partitions in that range?
or if there is a way of merging two daily partitions into one
r

rex

09/15/2022, 5:32 PM
I don’t believe
TimeWindowPartitionsDefinition
and its derivatives take in an end date - they currently return an infinite generator given the cron schedule and the start date. @sandy are there any work arounds for this? In the meantime, feel free to file a feature request for this
a

Alessandro Cantarelli

09/16/2022, 8:11 AM
The final goal I am drying to achieve is: If I partition on start day "x", I would like it to retrieve also the data from the day before. Something like a railing window partition.
What is the protocol for filing a new feature?
s

sandy

09/16/2022, 11:15 PM
You can achieve something like that with partition mappings. Here's an example someone posted a little while ago: https://dagster.slack.com/archives/C01U954MEER/p1658350429504999?thread_ts=1658345537.413359&cid=C01U954MEER. Would this work for you?
m

Matt Clarke

09/18/2022, 3:36 PM
Hi Sandy, Extension to this, are there any challenges which would arise if an asset was dependent on a past materialisation of itself? For example, to calculate something like a cumulative sum, the options would be: • Load all prior materialisations of the upstream asset • Load the new data in the latest partition of the upstream asset, plus the cumulative sum as of the previous materialisation of the cumsum asset (NDaysPartitionMapping(days = 1, offset = 1) of self)) Wondering whether or not this would violate the acyclic part of a DAG...in theory it is only ever reliant on a prior partition, so it isn't explicitly acyclic but could imagine this might lead to some issues!
a

Alessandro Cantarelli

09/20/2022, 8:30 AM
Thank you @sandy this was really helpful.
s

sandy

09/20/2022, 3:44 PM
hey @Matt - I missed this. I don't think an asset depending on a past materialization of itself currently works, but I'd like to support that in the future
m

Matt Clarke

09/20/2022, 4:12 PM
Ah, fair enough. Is there some workaround we could do in the meantime? We're currently just dumping to an external summary table but it means we end up with the potential for lots of fun weird states or race conditions. Fine while we are building out but probably not the most robust solution!
3 Views