What s the best practice for managing output files I was sta

Question

What s the best practice for managing output files I was starting to write a resource module for creating a directory on disk but wanted to consider the case where I m running on a distributed execution engine á la Dask Does the dagster aws S3 integration allow for that kind of situation out of the box

max · Answer

the s3 integration would def work for this

max · Answer

you might also be interested in exploring the `FileManager` system

max · Answer

which tbh is a little rough because it s still under exercised

max · Answer

but the context object you get in the body of a solid compute function has a `file manager` member

max · Answer

that you can use to manage files in a storage backend independent way

max · Answer

so that you could switch from using a local filesystem in test to s3 in prod etc

Tobias Macey · Answer

Good to know thanks