What's the "best practice" for managing output fil...
# announcements
What's the "best practice" for managing output files? I was starting to write a resource module for creating a directory on disk, but wanted to consider the case where I'm running on a distributed execution engine á la Dask. Does the dagster-aws S3 integration allow for that kind of situation out of the box?
the s3 integration would def work for this
you might also be interested in exploring the
which, tbh, is a little rough because it's still under-exercised
but the context object you get in the body of a solid compute function has a
that you can use to manage files in a storage backend-independent way
so that you could switch from using a local filesystem in test to s3 in prod, etc
Good to know, thanks!