is there a way to pass a list of file paths to an `Out` and

Question

is there a way to pass a list of file paths to an `Out` and have the `IOManager` copy the files to its storage location

Charlie Bini · Answer

scenario say my op pulls data from a table in 10000 row chunks saves each chunk to its own `jzon gz` file and results in 1 GB of total data I d like to avoid loading all of this in memory at once to pass to the out

jamie · Answer

I think writing a custom IO manager would be the only solution for this currently

Charlie Bini · Answer

my alternative would be to use the FileManager I m on GCS and just call it in the op

Charlie Bini · Answer

thanks for clarifying