ConfigurablePickledObjectS3IOManager on windows re...
# ask-community
ConfigurablePickledObjectS3IOManager on windows returns an invalid path? Hi there, I'm just switching from an old (0.13.x) version to 1.3.9 and trying to make use of Configurable resources and IO managers where possible. I configured ConfigurablePickledObjectS3IOManager with:
But the output went to the bucket root location with a key like: orchestration\alts\dev\storage\<run_id> I think this is because this method, is returning a WindowsPath which is not being handled correctly downstream, since by the time it reaches here it will not be transformed properly into an S3 key with forward slashes.
BTW, the crucial issue for us is that, without the object prefix being interpreted in a "folder-like" fashion, we can't apply bucket permission policies that allow our assets to materialise, since they need DELETE permissions, which we can't have granted at the bucket level but can request at a subfolder level.
For the time-being I've monkey-patched dagster_aws.s3.io_manager.PickledObjectS3IOManager.dump_to_path. Where it previously did this:
self.s3.upload_fileobj(pickled_obj_bytes, self.bucket, str(path))
it now does this:
self.s3.upload_fileobj(pickled_obj_bytes, self.bucket, str(<|>_posix()))