Hi! I have been using multi-partitioning for a sho...
# dagster-feedback
Hi! I have been using multi-partitioning for a short time and I find the concept very powerful for my use case. However, it often happens that I don't want to have a partition for each unique combination of dimensional partitions. For example in the case where I need to perform the same operation on several clients that have different time partitioning, I need to consider the oldest date to create all the keys. The consequence of this is that some of the keys are useless to me and return errors if I try to materialize them.
I imagined a system where it would be possible to use + and - operations to combine partitionings. For example something like this:
Copy code
partitions_def = MultiPartitionsDefinition(
        "date": DailyPartitionsDefinition(start_date="2022-01-01"),
        "color": StaticPartitionsDefinition(["red", "yellow"]),
) + MultiPartitionsDefinition(
        "date": DailyPartitionsDefinition(start_date="2023-01-01"),
        "color": StaticPartitionsDefinition(["blue", "green"]),
hi @Jordan! this is definitely an interesting use case (and a neat API!), although I think this would be fairly complex to implement. even just in terms of the UI, it's somewhat hard to imagine a clean way of representing the space of possible partitions to the user. another way of approaching this problem (although much less elegant) might be to have some special metadata that you attach to the assets, then only do things in your functions if you have a valid partition key combination, something like :
Copy code
    "red": "2022-01-01",
    "yellow": "2022-01-01",
    "blue": "2023-01-01",
    "green": "2023-01-01",

    metadata={"start_date_mapping": START_DATES_BY_COLOR},
def my_asset(context):
    # pseudocode
    color = context.partition_key.color_dimension
    start_date = context.metadata[start_date_mapping][color]
    if context.partition_key.time_dimension >= start_date:
        ... # do stuff
        yield Output(...)