Do assets need to be able to fit into memory? I'm guessing practically they do and that partitioning is the recommended path to avoid assets which are too large for in memory processing?

No, you can use

DynamicOut

for this

I just wrote up slightly longer answer for this here: https://github.com/dagster-io/dagster/discussions/11387.

Thank you for the detailed response, reading this now!