https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Dennis Torres

08/30/2023, 10:22 AM
Hello! I have two dynamic partitions that are hierarchical like the following: Parent Partition keys:
Copy code
A, B, C
Child Partition keys:
Copy code
A_1, A_2, A_3, A_4, B_1, B_2, C_1, C_2, C_3
I’d like to have a downstream asset with a Parent Partition be able to depend on an upstream asset with a Child Partition. The intent is to have each Parent combine the data of its children. Something like this:
Copy code
@asset(partitions_def=child_partition)
def child_data(context):
    parent_id, child_id = context.partition_key.split("_")  # e.g. "A_1" → ("A", "1")
    # …

@asset(
    partitions_def=parent_partition,
    ins={
        "child_data": AssetIn(
            partition_mapping=CustomPartitionMapping(
                get_downstream_key_from_upstream_key=lambda key : key.split("_")[0]  # e.g. "A_1" → "A"
            ),
        )
    },
)
def parent_data(context, child_data):
    parent_id = context.partition_key
    children = child_data.values()
    # …
I made up the
CustomPartitionMapping
class, so the above won’t work 😅. Is there a good strategy for going about this? Unfortunately, the partition keys are dynamic because they aren’t known ahead of time. After looking through the partition mapping classes, it seems like using dynamic partition keys makes this a challenge. Any help is very much appreciated, Thanks!
c

claire

08/30/2023, 8:32 PM
Hi Dennis. Do your child partition keys contain every unique combination of parent partition keys and [1, 2, 3, 4...]? If so, you could represent the child partitions def as a
MultiPartitionsDefinition
with two dynamic dimensions ([a, b, c] and [1, 2, 3, 4...])
d

Dennis Torres

08/31/2023, 7:21 PM
Thanks, I’ll try that out. There will be lots of combinations that don’t exist because parents have a varying number of children.