is there anyway to set the endpoint for s3 storage? I'm using minio in my cluster as a gateway/abstraction. Something like...

config:
  s3_bucket: STORAGE_BUCKET
  endpoint: MINIO-SERVICE-NAME:PORT

Yep - https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/s3/compute_log_manager.py#L35

thanks, I'll verify if this works tomorrow and look into the

verify

settings too

I fixed my verify problems by upgrading to the latest version of `certifi` It seems that if you have

certifi

installed; Boto will use its certs rather than your system certs (at least on a Mac); potentially giving different behaviour when you curl an endpoint vs access it via Boto.

we solved it by adding

verify=False

to the

utils.py

in dagster_aws/s3/utils.py. I guess would be nice to have it build in as config. By the way, are you using spark? I’m struggling with io manger and spark and dataframes (pickles are not possible). Need to write my own io-manager.