https://dagster.io/ logo
#ask-community
Title
# ask-community
v

Vinnie

04/13/2023, 1:12 PM
Is it possible to have assets with different partitions as part of the same multi asset? Might sound a little convoluted, but I have a
multi_asset
that’s supposed to process a few csv files. 2/4 of these files are present in both partitions, plus one specific for each. Simplified, it looks like the following:
Copy code
@multi_asset(
    outs={
        "some_asset": AssetOut(), # should include both a and b as static partitions and all dynamic
        "some_other_asset": AssetOut(), # should include both a and b as static partitions and all dynamic
        "a_asset": AssetOut(is_required=False), # should either only include dynamic partitions, or only "a" and dynamic
        "b_asset": AssetOut(is_required=False), # should either only include dynamic partitions, or only "b" and dynamic
    },
    partitions_def=MultiPartitionsDefinition(
        {
            "static": StaticPartitionsDefinition(["a", "b"]),
            "dynamic": DynamicPartitionsDefinition(name="my_dynamic_partition"),
        }
    ),
    can_subset=True,
)
currently the workaround is to just return
None
if the asset is not expected to exist for the static partition called, but it would be great for observability if dagster could know that some of the assets need to be partitioned differently
c

claire

04/13/2023, 7:42 PM
Hi Vinnie. Unfortunately this is not currently possible, we enforce that each partitions def in a multi asset is the same. But I think it's reasonable and doable for us to support this behavior--it's likely just something that we haven't had the chance to implement yet. Would you mind opening an issue for this?
v

Vinnie

04/13/2023, 8:30 PM
11 Views