what’s the logic to decide which partition becomes primary_dimension in MultiPartitionsDefinition?

The

MultiPartitionsDefinition

in Dagster does not have a concept of a "primary_dimension" as such. Instead, it allows you to define multiple dimensions for partitioning an asset, and each dimension is defined by its own

PartitionsDefinition

. The total set of partitions will be the cross-product of the partitions from each

PartitionsDefinition

. When you define a

MultiPartitionsDefinition

, you provide a mapping of dimension names to their respective

PartitionsDefinition

. The order in which you define these dimensions does not designate any of them as primary; they are simply used to create a partition for each unique combination of dimension partitions. Here is an example from the documentation:

python
from dagster import (
    AssetExecutionContext,
    DailyPartitionsDefinition,
    MultiPartitionsDefinition,
    StaticPartitionsDefinition,
    asset,
)

@asset(
    partitions_def=MultiPartitionsDefinition(
        {
            "date": DailyPartitionsDefinition(start_date="2022-01-01"),
            "color": StaticPartitionsDefinition(["red", "yellow", "blue"]),
        }
    )
)
def multi_partitions_asset(context: AssetExecutionContext):
    if isinstance(context.partition_key, MultiPartitionKey):
        <http://context.log.info|context.log.info>(context.partition_key.keys_by_dimension)

In this example, the asset would contain a partition for each combination of color and date, such as

red|2022-01-01

,

yellow|2022-01-01

,

blue|2022-01-01

, and so on. There is no indication that one dimension is more important or "primary" than the other. If you need to define a dependency relationship between a multi-partitioned asset and a single-dimension partitioned asset, Dagster provides mechanisms like

MultiPartitionMapping

to handle such cases, but this does not imply the concept of a primary dimension within the

MultiPartitionsDefinition

itself. Partitioning assets | Dagster button Dagster Docs button PartitionMapping between multi-dimension... button Sparse MultiPartition Materialization Se... button Thank you for your feedback!

@Scout partition definition has a primary_condition

I apologize for any confusion earlier. As of my knowledge cutoff in early 2023, Dagster's

MultiPartitionsDefinition

does not have a built-in concept of a

primary_dimension

or

primary_condition

as part of its API. The

MultiPartitionsDefinition

is used to define multiple dimensions of partitioning, and each dimension is treated as a separate axis for partitioning without any inherent prioritization or primary status. If there has been an update to Dagster that introduces a

primary_condition

or similar concept to the

MultiPartitionsDefinition

that I am not aware of, I would recommend checking the latest Dagster documentation or release notes for the most up-to-date information on this feature. In the context of partition mappings, such as when defining dependencies between assets with different partitioning schemes, you would specify how partitions relate to each other using a

PartitionMapping

. This can involve specifying which dimension in a multi-dimensional partition corresponds to a single-dimensional partition, but this is about defining relationships for the purpose of dependency resolution rather than designating a primary partition. If you are working with a specific API or feature in Dagster that mentions a

primary_condition

and it is not covered in the documentation I have access to, please provide more context or refer to the latest documentation for guidance on how to use that feature. partition_key argument to materialize an... button Partition | Dagster Glossary button Allow filtering by only one dimension of... button How to depend on only one specific key f... button Thank you for your feedback!

@Scout this is the code link: https://github.com/dagster-io/dagster/blob/8d2a7d2541b9cf873ee55e0eefc0049e201d9bf[…]gster/dagster/_core/definitions/multi_dimensional_partitions.py

Thank you for providing the link to the Dagster codebase. However, as a friendly developer relations assistant for Dagster, I am unable to directly access external content such as GitHub repositories or other websites. My responses are based on the information available up to my last update in early 2023. If you're referring to a specific feature or code implementation in the Dagster codebase that is not covered in the documentation available to me, I would recommend exploring the code directly or reaching out to the Dagster community for more detailed insights. The Dagster community can be a valuable resource for understanding recent changes and features that may not be covered in my current knowledge base. For the most accurate and up-to-date information regarding

primary_condition

or any other specific implementation details, please consult the latest version of the Dagster documentation or the codebase itself. If you have any other questions or need further assistance with Dagster concepts that are within my knowledge, feel free to ask! Making which partition mappings are bein... button Dagster Docs button Partitioning assets | Dagster button PartitionMapping between multi-dimension... button Thank you for your feedback!