https://dagster.io/ logo
m

max

03/29/2019, 10:32 PM
i wonder if it really needs to be context-aware. we could pass a Filelike (binary stream) around, and let the intermediates manager do all the writing to/from disk, and then also expose like, context.storage, which could tell you what it was (context.storage.storage_mode) and also give you Paths (context.storage.put_object() -> Path)