Is there a comprehensive list of available intermediate storage definitions? I am building a distributed pipeline and the s3 intermediate from

dagster_aws

won't be a valid option in production.

Hi Andy, what is your use case?

We are pushing pipelines through dask as an intermediate for a slurm cluster. In order to push using distributed, I have to supply a ModeDefinition which provides an intermediate storage definition, and it complains and fails if I use the filesystem intermediate. I am OK to use s3 in development, but that won't be an option when we hit production.

what would the production be using?

That's what we're not sure of yet. I was hoping to find a list of intermediates documentation so that we could make a decision about how we would proceed there.

i see. sorry the docs for that part aren’t finished yet. we currently have the following • gcs_intermediate_storage https://github.com/dagster-io/dagster/blob/4a91c9d09b50db93e9174c93a4ada0e138e3a046/python_modules/libraries/dagster-gcp/dagster_gcp/gcs/system_storage.py#L24 • adls2_intermediate_storage https://github.com/dagster-io/dagster/blob/033e51a86eff54b1d4289a984c711a17f13e6041/python_modules/libraries/dagster-azure/dagster_azure/adls2/system_storage.py#L24 • s3_intermediate_storage https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/system_storage.py

Hey Andy! Just curious: why doesn’t s3 fits your production needs? And how about a self-hosted s3?

Thanks @yuhan, much appreciated.

We used minio (https://github.com/minio/minio) and zenko (https://github.com/scality/cloudserver) - both compatible with dagster_aws, self-deployed containerized solutions. Though minio is nicer with its gui, it is 50x slower with dagster_aws compute_logs (https://github.com/dagster-io/dagster/issues/2438) due to some strange boto3 behaviour. So we’ve switched to zenko for now

Awesome, thanks @matas -- I'll check this out