https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
a

Alec Koumjian

03/09/2023, 4:32 AM
We have to go through a lot of hoops right now to roll up our daily partitioned data into monthly partitioned data. Normally, we can just declare a downstream asset with the monthly partitioning and the default partition mapping will make it all happen. However, we have multiple datasets that are not continuously updated (they ended in the past). This means we have to use a
StaticPartitionsDefinition
and we can't do easy mapping. Instead we have to send it into an
op
through a
job
which has to emit the
AssetMaterializations
manually after also doing the I/O manually and then we also have to declare the downstream asset as a
SourceAsset
so we can properly feed it into the graph. All that is to say, what would be the lift to support
TimeWindowPartitionsDefinition
with an
end
? Either that, or figuring out a way to support custom mappings.
a

Adam Bloom

03/09/2023, 4:47 AM
a

Alec Koumjian

03/09/2023, 11:48 AM
Hah, my coworker already commented on that issue. Thank you
s

sandy

03/09/2023, 8:09 PM
It basically requires updating TimeWindowPartitionsDefinition to acknowledge the end date. We don't currently have the bandwidth to take this on, but I could review and provide tips if it's something you'd be able to submit a PR for?
a

Alec Koumjian

03/10/2023, 4:51 PM
Yes, I added to the existing ticket. We could look at this if we had some general guidance on what not to do.
s

sandy

03/10/2023, 5:03 PM
I think a useful heuristic is that changes need to be made to everywhere inside
TimeWindowPartitionsDefinition
that's handling
self.end_offset
I can also answer more specific questions if it would be helpful
a

Alec Koumjian

03/10/2023, 5:28 PM
Sure, makes sense. We might dig into this maybe next week? Thanks
👍 1