Hi Team, Not sure if I have understood MultiParti...
# ask-community
m
Hi Team, Not sure if I have understood MultiPartitionsDefinitions correctly, would the following always produce the cross product of the two dimensions?
Copy code
@asset(
    partitions_def=MultiPartitionsDefinition(
        {
            "customer_code": StaticPartitionsDefinition(["abc","youi"]),
            "datasource_name": StaticPartitionsDefinition(["tv", "search"]),
        }
    )
)
So the above would create the following partitions: ['abc|tv', 'abc|search', 'youi|tv', 'youi|search']. But this may not be the case for us, each customer can have a different list of data sources. Is there a way to code a mapping such that is produces: ['abc|tv', 'abc|search', 'youi|tv']. i.e. 'youi' does not have the 'search' datasource.
s
While a multipartitionsdefinition is the cleanest way to go when you want the cross-product, you can use a StaticPartitionsDefinition and create the strings that you like. For example, you could create: • abc|tv • abc|search • youi|tv And then in your asset/op definition, split manually and operate on them.
Copy code
customer, datasource = partition_key.split('|')
# perform some work that depends on customer and datasource
....
This will work just fine. You'll just see the single partition key in the dagit UI, though.
j
hi @Manan P that’s correct, the multipartitions only supports cross product