https://dagster.io/ logo
Join Slack
Powered by
Hi :slightly_smiling_face: I have an asset with `M...
# ask-community
r
Hi 🙂 I have an asset with 
MultiPartitionsDefinition
combined of an hourly one and a dynamic one. I know it’s possible to run backfill as one single job for many partitions, as demonstrated during the last community meeting D Is it possible to combine many partitions as well for schedules (and the underlying jobs)?
c
Hi Roei. Here's an example of how to backfill a range of partitions in a single run via a schedule: https://github.com/dagster-io/dagster/discussions/11653#discussioncomment-5493977
A valid range is any contiguous subset of keys in 
partitions_def.get_partition_keys
, and for multipartitioned assets this gets a little complicated. I.e.:
Copy code
MultiPartitionsDefinition(
    {
        "date": DailyPartitionsDefinition(start_date="2022-06-11"),
        "abc": StaticPartitionsDefinition(["a", "b", "c"]),
    }
)
has partition keys 
['a|2022-06-11'...'a|2023-01-16', 'b|2022-06-11'...'b|2023-01-16', 'c|2022-06-11'...'c|2023-01-16']
so selecting a range 
a|2022-06-11...b|2022-06-11
will select all of the "a" partition dimension keys. So basically out of the partition keys you want to backfill, you'll have to determine which contiguous ranges exist and kick off a run request per range.
r
Thanks @claire! Can I do the same while the 
date
partition is fixed and the 
abc
partition in your example is the “range”? i.e 
['a|2022-06-11','b|2023-01-11','c|2022-06-11']
c
unfortunately not, the partition keys are in a fixed order as defined by 
get_partition_keys
, so 
['a|2022-06-11'...,'c|2022-06-11']
includes all the keys in between in the 
get_partition_keys
list. I know that this is awkward, ideally we should be able to select ranges across any arbitrary dimension.
d
Is this answer still the state of multipartition keys? I am running up against wishing I could do something like 
start = context.asset_partition_key_range_for_output().start
for a multipartitioned asset with an hourly partition
5 Views
Open in Slack
PreviousNext
Powered by