The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

is there a way to pass a list of file paths to an `Out` and have the `IOManager` copy the files to its storage location?

scenario: say my op pulls data from a table in 10000 row chunks, saves each chunk to its own `jzon.gz` file, and results in 1 GB of total data. I'd like to avoid loading all of this in-memory at once to pass to the out

I think writing a custom IO manager would be the only solution for this currently

my alternative would be to use the FileManager (I'm on GCS) and just call it in the op