Hello I have a question request for `class MultiPartitionsDe dagster #ask-community

Hello, I have a question/request for `class MultiP...

Jeff

01/13/2023, 5:16 PM

Hello, I have a question/request for

class MultiPartitionsDefinition(PartitionsDefinition)

in 🧵

Jeff

01/13/2023, 5:17 PM

Is there a way to set the order or the partitions? Right now I see it’s sorted:

Copy code

self._partitions_defs: List[PartitionDimensionDefinition] = sorted(
            [
                PartitionDimensionDefinition(name, partitions_def)
                for name, partitions_def in partitions_defs.items()
            ],
            key=lambda x: x.name,
        )

I want to be able to use something like

OrderedDict

instead of forcing alphabetical order.

owen

01/13/2023, 6:08 PM

cc @claire

claire

01/13/2023, 7:35 PM

Hi Jeff, unfortunately not. We use the alphabetical ordering to enforce that the serialized partition keys e.g.

dim_1_key|dim_2_key

remain in the same format no matter the provided order of the partition dimensions. Out of curiosity, what's the reason you'd like to set the order of the partitions?

Jeff

01/13/2023, 7:39 PM

I’m using a partition mapping to get upstream assets. Since the order

dim_1_key|dim_2_key

may not be the same order as specified in the

MultiPartitionsDefinition

this is a bit annoying. I think allowing

OrderedDict

to specify the order would be helpful. Additionally, when specifying the partition for runs, I have date as one partition and some category for the other partition (this category varies between jobs/assets). To I would like to have date always be the second partition as this keeps it consistent between jobs/assets and allows for a logical ordering of partitions based on scope/importance.

Jeff

01/13/2023, 7:46 PM

Part of the issue is that

PartitionMapping.get_upstream_partitions_for_partition_range

is passed a string as the partition for

downstream_partition_key_range

, if this was a dict or some sort of partition object, this would be more clear. Right now I have to parse the string and getting the keys by position is not the cleanest.

claire

01/13/2023, 7:50 PM

Hm... so I think the string object in the partition key range is actually a

MultiPartitionKey

object (which subclasses string). There should be a

multi_partition_key.keys_by_dimension

field which should return a mapping from dimension name -> partition key for that dimension

Jeff

01/13/2023, 7:57 PM

Thanks, I’ll look into that. I think this resolves most of the issues I have. I guess the only thing left that I can think of is that for selecting the partition in dagit, the user may want some way to specify the order of the partitions to maintain consistency between assets/jobs. This is more of a QOL thing though.

claire

01/13/2023, 8:42 PM

Definitely--we could think of some way to select an ordering of partitions in the multidimensional view

ty spinny 1

11 Views

Open in Slack

Previous Next