Hi all, I have a partitioned multi-asset where som...
# ask-community
Hi all, I have a partitioned multi-asset where some of the outputs are optional (essentially some of the files are missing for some of the partitions — that’s fine in this case). A non-partitioned downstream asset should consume one of the optional outputs of the upstream multi-asset (when it’s available), but I get
errors as the downstream asset seems to look for the missing outputs in the incomplete partitions. I would have expected them to simply be missing from the partition dict or have
as value. I can return a “dummy” value if the upstream file is missing and check for it in the downstream asset, but that would kinda defeat the purpose of an optional asset out. Am I missing something here or is this really not doable?
This probably leads to an "optional partition" concept that might not make sense... 🤔
Hi Andras - the default IO manager doesn't support the behavior you're talking about, but you could write your own IO manager (perhaps copy/paste the default IO manager- I can send you a code pointer if helpful) that ignores missing files instead of erroring on them Also, feel free to file a Github issue for adding options to the default IO manager to support this pattern
👍 1
Well, it turns out it’s an existing feature already with the
flag in metadata: https://docs.dagster.io/concepts/io-management/io-managers#custom-filesystem-based-io-manager Thanks for pointing me the right direction, I’m sure I couldn’t have found this.
Oops - I missed that!