https://dagster.io/ logo
#ask-community
Title
# ask-community
b

Bryce Arden

06/10/2023, 4:54 AM
Hey folks! Has successfully set an
endpoint_url
override with the UPathIOManager? Or better yet, share an existing s3 resource with a UPathIOManager using a
ResourceDependency
?
For additional context, I’m trying to use the example S3ParquetIOManager with a localstack s3 bucket, which creates a bucket for you with endpoint url
<http://localhost:4566>
, and access/secret keys of ‘test’/‘test’ by default. For some reason, the
endpoint_url
override doesn’t seem to be applying.
Copy code
base_path = UPath(self.base_path, endpoint_url=self.endpoint_url)
I’ve verified that I can connect to this bucket and write successfully with the
ConfigurablePickedObjectS3IOManager
using the same creds, which has led me to believe I’m probably doing something silly with UPath. I’m looking forward to the help!
Fwiw, I got this to work by putting the following in a
~/.config/fsspec/s3.json
Copy code
❯ cat ~/.config/fsspec/s3.json -p
{
  "s3": {
    "client_kwargs": {"endpoint_url": "<http://localhost:4566>"}
  }
}
But this doesn’t work very well with the resource system
2 Views