https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jeff

01/13/2023, 5:16 PM
Hello, I have a question/request for
class MultiPartitionsDefinition(PartitionsDefinition)
in 🧵
Is there a way to set the order or the partitions? Right now I see it’s sorted:
Copy code
self._partitions_defs: List[PartitionDimensionDefinition] = sorted(
            [
                PartitionDimensionDefinition(name, partitions_def)
                for name, partitions_def in partitions_defs.items()
            ],
            key=lambda x: x.name,
        )
I want to be able to use something like
OrderedDict
instead of forcing alphabetical order.
o

owen

01/13/2023, 6:08 PM
cc @claire
c

claire

01/13/2023, 7:35 PM
Hi Jeff, unfortunately not. We use the alphabetical ordering to enforce that the serialized partition keys e.g.
dim_1_key|dim_2_key
remain in the same format no matter the provided order of the partition dimensions. Out of curiosity, what's the reason you'd like to set the order of the partitions?
j

Jeff

01/13/2023, 7:39 PM
I’m using a partition mapping to get upstream assets. Since the order
dim_1_key|dim_2_key
may not be the same order as specified in the
MultiPartitionsDefinition
this is a bit annoying. I think allowing
OrderedDict
to specify the order would be helpful. Additionally, when specifying the partition for runs, I have date as one partition and some category for the other partition (this category varies between jobs/assets). To I would like to have date always be the second partition as this keeps it consistent between jobs/assets and allows for a logical ordering of partitions based on scope/importance.
Part of the issue is that
PartitionMapping.get_upstream_partitions_for_partition_range
is passed a string as the partition for
downstream_partition_key_range
, if this was a dict or some sort of partition object, this would be more clear. Right now I have to parse the string and getting the keys by position is not the cleanest.
c

claire

01/13/2023, 7:50 PM
Hm... so I think the string object in the partition key range is actually a
MultiPartitionKey
object (which subclasses string). There should be a
multi_partition_key.keys_by_dimension
field which should return a mapping from dimension name -> partition key for that dimension
j

Jeff

01/13/2023, 7:57 PM
Thanks, I’ll look into that. I think this resolves most of the issues I have. I guess the only thing left that I can think of is that for selecting the partition in dagit, the user may want some way to specify the order of the partitions to maintain consistency between assets/jobs. This is more of a QOL thing though.
c

claire

01/13/2023, 8:42 PM
Definitely--we could think of some way to select an ordering of partitions in the multidimensional view
ty spinny 1
11 Views