https://dagster.io/ logo
#ask-ai
Title
# ask-ai
m

Marcel Steinbach

03/06/2024, 3:02 PM
@Scout how do I map a multi-dimensional partitioned asset to a single dimension partitioned asset, where second asset's partition is uses only the primary partition of the first asset?
s

Scout

03/06/2024, 3:02 PM
To map a multi-dimensional partitioned asset to a single-dimension partitioned asset, where the second asset uses only the primary partition of the first asset, you can use the
MultiToSingleDimensionPartitionMapping
provided by Dagster. This partition mapping assumes that any multi-partition key with the specified dimension in the selected dimension is a dependency for the single-dimension partitioned asset. Here's an example from the documentation that illustrates how you might define such a relationship:
Copy code
python from dagster import (     DynamicPartitionsDefinition,     WeeklyPartitionsDefinition,     asset,     MultiPartitionsDefinition, )  projects_partitions_def = DynamicPartitionsDefinition(name="project") # a, b, c  @asset(     partitions_def=MultiPartitionsDefinition(         partitions_defs={             "week": WeeklyPartitionsDefinition("2023-01-01"),             "project": projects_partitions_def,         }     ) ) def upstream():     ...  @asset(partitions_def=projects_partitions_def) def downstream(context, upstream):     ...
In this example, when you execute the
downstream
asset with a specific partition (e.g., partition "a"), it will load every partition of the
upstream
asset that has "a" as its project dimension:
a|2023-01-01
,
a|2023-01-08
, etc. This is because the
projects_partitions_def
is used as the partitions definition for both the multi-partitioned
upstream
asset and the single-dimension
downstream
asset. Dagster automatically uses the
MultiToSingleDimensionPartitionMapping
to map the dependencies correctly. If you need to customize the partition mapping further, you can define your own partition mapping class by subclassing
PartitionMapping
and implementing the necessary methods to specify the exact mapping logic you require. However, the example above should suffice for the case where you want to map the primary dimension of a multi-dimensional partitioned asset to a single-dimension partitioned asset.
6 Views