Brendan Jackson
08/22/2023, 5:42 PMOutputContext
- has_asset_partitions
and has_partition_key
- that are subtly different, and caught me out in their different behaviour.
When I materialise a single partition, has_asset_partitions
is true, as is has_partition_key
.
When I materialise multiple partitions in a single run (using the fact it's a time range), has_asset_partitions
is true, but has partition_key
is false.
This was a bit of a surprise, and I wonder if has_partition_key
name or at least doc needs to reflect this.
"""Whether the current run is a partitioned run."""
I'd have expected multiple partitions in a single run to count as a partitioned run.Pedram Navid
08/23/2023, 2:55 AMhas partition_key
is only true for a single partition run, not a backfill of multiple partitions.
2. In has_partition_key
reference has_asset_partitions
as the right property if trying to detect a partitioned asset.
Does that sound right?