Hey Team! We have a use case with a daily partitioned asset that has a daily partitioned upstream dependency. We are looking for a partition mapping such as the one described in this comment. The downstream's single daily partition depends on all partitions of the upstream asset. Two questions: • Is there already a Github issue tracking the implementation/idea of the

PrecedingPartitionMapping

? Are there any plans underway to implement the mapping described in that comment? • Any ideas for alternative solutions for mapping a single partition of one asset to all partitions of an upstream asset? Please let me know if any additional details or information is needed to help answer the questions. Thanks!

Maybe this type of partition mapping would be helpful: https://docs.dagster.io/_apidocs/partitions#dagster.AllPartitionMapping

Thanks for the link Vitaly, but that isn’t quite what we are looking for. That maps all downstream partitions to all upstream partitions. We need 1 downstream partition mapped to all upstream partitions (regardless are partition granularity).

I suspect you might be able to implement custom mapping logic by inheriting PartitionMapping class and implementing this logic manually. It should be straightforward .

That seems like a reasonable idea, but I thought I remembered reading something in the Dagster docs that inheriting from classes was not recommended and could break in future releases. Is that not the case?

In theory, there is always a risk of backwards incompatibility. But it should not stop you from customising things. If you pin Dagster version to ~=1.3 , it should be safe for production.

hi @Jesse Cochran! two things: first -- (and apologies for this info being non-obvious!) I would not recommend subclassing PartitionsDefinition, as these partitions need to be able to survive a serde layer. Essentially, many functions (such as rendering pages in the UI or launching backfills) require being able to operate over your asset graph without having to invoke any user code. This model breaks apart when custom PartitionsDefintion classes are defined. for the original question, it sounds like you could still use a TimeWindowPartitionMapping for this usecase, as this supports an end_offset parameter. If you were to set this to an arbitrarily high number, then any partition for the downstream asset would depend on all upstream partitions that happened before that time. would this work in your case?

Hi @owen - thanks for the response! • Appreciate the clarification on the PartitionDefinition subclassing. Great to know! • Would this work for setting

start_offset

for the

TimeWindowPartitionMapping

to an arbitrarily large negative number? The documentation states if you set

end_offset

to say 1, then the downstream partition

2022-07-04

would map to

2022-07-04

and

2022-07-05

. I think we would want the opposite - if the most recent partition is

2022-07-04

in the downstream, we want this to map to upstreams

2022-07-04

,

2022-07-03

,

2022-07-02

, ....and every partition that has existed up until the current partition. ◦ Want to confirm this want cause an error? Once we get to the earliest partition, lets say this partition is

N

, of the upstream partition window, it wouldn't throw an error saying it couldn't find the partition at

N - 1

- day before earliest partition?

ah you're totally right got the wires crossed in my head.

start_offset

is the way to go. and the time window partition mapping will handle the boundary case, and won't try to select partitions that don't exist

Awesome thats great news. Thanks for the confirmation and help Owen - much appreciated!

no problem :blob_salute: