Hi all, I have a partitioned multi-asset where some of the outputs are optional (essentially some of the files are missing for some of the partitions — that’s fine in this case). A non-partitioned downstream asset should consume one of the optional outputs of the upstream multi-asset (when it’s available), but I get

FileNotFoundError

errors as the downstream asset seems to look for the missing outputs in the incomplete partitions. I would have expected them to simply be missing from the partition dict or have

None

as value. I can return a “dummy” value if the upstream file is missing and check for it in the downstream asset, but that would kinda defeat the purpose of an optional asset out. Am I missing something here or is this really not doable?

Hi Andras - the default IO manager doesn't support the behavior you're talking about, but you could write your own IO manager (perhaps copy/paste the default IO manager- I can send you a code pointer if helpful) that ignores missing files instead of erroring on them Also, feel free to file a Github issue for adding options to the default IO manager to support this pattern

Well, it turns out it’s an existing feature already with the

allow_missing_partitions

flag in metadata: https://docs.dagster.io/concepts/io-management/io-managers#custom-filesystem-based-io-manager Thanks for pointing me the right direction, I’m sure I couldn’t have found this.

Oops - I missed that!