R Lucas
02/20/2024, 7:52 AMScout
02/20/2024, 7:52 AMPartitionMapping
that combines both TimeWindowPartitionMapping
and LastPartitionMapping
to map the last 10 hourly partitions of a parent asset and materialize an unpartitioned child asset. The TimeWindowPartitionMapping
allows you to map partitions based on a time window relative to the downstream partitions, and the LastPartitionMapping
maps to the last partition of the upstream asset. However, the documentation does mention that you can use TimeWindowPartitionMapping
with start_offset
and end_offset
to map partitions from the upstream asset to the downstream asset. For example, if you want to map the downstream partition “2022-07-04” to the upstream partitions “2022-07-03” and “2022-07-04”, you can set start_offset=-1
and end_offset=0
. For your specific use case, you might consider implementing a custom logic outside of the provided PartitionMapping
options. This could involve creating a custom sensor or some other mechanism to trigger the materialization of the unpartitioned child asset based on the completion of the last 10 hourly partitions of the parent asset. If you need to materialize an unpartitioned asset based on the state of partitioned upstream assets, you may need to write custom logic to handle this, as the existing PartitionMapping
options may not cover this specific scenario. The documentation does not provide an example of a PartitionMapping
that directly supports this use case.