Hey, I’m having an issue using the s3_fake_resourc...
# ask-community
Hey, I’m having an issue using the s3_fake_resource, with the following error
Copy code
def put_object(self, Bucket, Key, Body, *args, **kwargs):
        self.mock_extras.put_object(*args, **kwargs)
>       self.buckets[Bucket][Key] = Body.read()
E       AttributeError: 'bytes' object has no attribute 'read'
I’m simulating mocking just uploading a zipped file so just some bytes, but it seems the fake resource expect to be able to
the Body? Why is this? Or am I using the resource wrong?
👍 1
It looks like the S3 PutObjects API expects:
bytes or seekable file-like object
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_object But the
incorrectly implements it to only take a seekable file-like object. We can get a fix out for this - sorry about that!
Hey, thanks for the quick reply @jordan! No worries, hopefully it’s a quick easy fix 🙂
also I noticed function
does not have the
decorator, meaning I can’t use it directly as a dagster resource - should it?
The deeper I dig on this the less polished it gets - it looks like perhaps when it was first introduced, it was intended to be used with `ResourceDefinition.hardcoded_resource()`: https://github.com/dagster-io/dagster/blob/6ee348c9b5645688c1f3de3521158a8ebf3c1b09/python_modules/libraries/dagster-aws/dagster_aws_tests/s3_tests/test_s3_file_manager.py#L141-L152 And over time, it’s been both deprecated and then re-added - but never brought under test or up to date with current Dagster idioms. It’s actually not used anywhere in Dagster’s own tests anymore - instead, we use a real S3 resource but mock out the S3 service with moto: https://github.com/spulec/moto You might find this to be a less brittle approach since the moto implementations (particularly around popular services like S3) are quite well supported.
ok makes sense. Would something like the following be a good way to create a mock s3 resource in that case?:
Copy code
import boto3
from moto import mock_s3
from dagster import resource, Array

@resource(config_schema={"buckets": Array(str)})
def s3_resource_mock(init_context):
    conn = boto3.resource("s3", region_name="eu-west-2")

    for b in init_context.resource_config["buckets"]:

    return conn
being a list of buckets I expect to find in s3
ok nvm I see I just need to use the normal s3_resource but decorate my test functions. Works as needed thanks 🙂
👍 1