https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
b

Brendan Jackson

08/22/2023, 5:42 PM
Context: I'm using the output context to detect if an asset is partitioned, and if so using that to delete rows in a database corresponding to the partition (i.e. between a start and end date) prior to insertion in order to enforce idempotency. There's a couple of methods on the
OutputContext
-
has_asset_partitions
and
has_partition_key
- that are subtly different, and caught me out in their different behaviour. When I materialise a single partition,
has_asset_partitions
is true, as is
has_partition_key
. When I materialise multiple partitions in a single run (using the fact it's a time range),
has_asset_partitions
is true, but
has partition_key
is false. This was a bit of a surprise, and I wonder if
has_partition_key
name or at least doc needs to reflect this.
Copy code
"""Whether the current run is a partitioned run."""
I'd have expected multiple partitions in a single run to count as a partitioned run.
p

Pedram Navid

08/23/2023, 2:55 AM
thanks for that feedback! that's very subtle indeed. I think two improvements could be: 1. Clarify that
has partition_key
is only true for a single partition run, not a backfill of multiple partitions. 2. In
has_partition_key
reference
has_asset_partitions
as the right property if trying to detect a partitioned asset. Does that sound right?
👍 1
2 Views