Hi, I can't find this question asked and answered ...
# ask-community
p
Hi, I can't find this question asked and answered anywhere. Our case is an asset that uses two (or more) different partition definitions. We would like: 1. A fairly wide date partition (perhaps monthly) for initial backfills. This is just to get our full historical data loaded. 2. A daily partition for day-to-day incremental growth in the asset. The code is the same - only the size of the eventual dataset should change. I can't see in the documentation how such a case should be handled. However, we'll need to deal with these things: • The external resource we fetch from will need to have a method (or methods) that can deal with either partition type • The IOManager will need to write the partition_key and an indicator as to the partition type into the asset output, so that we can deal with the case where we need to replace a partition (delete/insert) • We need to know at runtime which partition type (e.g. Monthly or Daily) we're dealing with so that the partition_key we write back to the output can be interpreted accordingly, assuming both Monthly and Daily might produce the same date format I can see I can get the partition_key and partition_def type at run time. But is there a good way to have an asset that can be materialized by either of two partition_defs?
c
Hi Paul. There isn't a great way to have an asset that can be materialized by multiple partitions defs. In Dagster, materializations are serialized as events with a partition key, and it's possible for a monthly partition and daily partition to have the same partition key (i.e.
2023-01-01
). This means that it's not currently possible to identify whether a materialization with a
2023-01-01
key materialized for the whole month or just one day. If you primarily want to be able to have large partitions for more efficient backfills, I recommend you instead use single run backfills: https://docs.dagster.io/concepts/partitions-schedules-sensors/backfills#single-run-backfills Using single run backfills, a single run will be kicked off for each partition range, and assuming your IO manager can operate on partition ranges, a single output will be persisted.