How can i `group` dynamic partitions which are adj...
# ask-community
How can i
dynamic partitions which are adjacent event timestamps into a downstream model with a time-based partition such as @daily or @weekly ? I have a source (S1) that is being refreshed as-needed during the day (dynamic partitions as timestamps) and I want the downstream dependency to run a daily report (R1) using a time-based partition @daily. So for a given day, the R1 partition depends on all the S1 partitions in that day range.
hi @Johno Scott! this is an interesting use case, but this sort of dynamic partition mapping is not currently supported. would you mind filing an issue for this? cc @claire who I believe was thinking about something similar
Hi Johno! Here's a link to an issue that more generally explores custom mappings for dynamic partitions: Some thoughts we've played around with: • having some sort of regex function to map partitions to each other • having some sort of yaml file that defines the partitions that are mapped to one another Though for your use case it might make sense to have a function defined in dagster that basically converts a timestamp into a time partition window
@owen I will file an issue.
@claire thanks I will check that out