Ian Macleod
02/20/2024, 9:39 PM'S3Resource' object has no attribute 'get_object'
for my dagster s3 resource. I am trying to retrieve a file from s3 given an s3 uri. What's the best way to do this with the dagster s3resource objectScout
02/20/2024, 9:39 PMS3Resource
object in Dagster provides a way to interact with AWS S3. However, the S3Resource
object itself does not have a get_object
method. Instead, you should use the get_client()
method provided by the S3Resource
to obtain a boto3 S3 client, which does have the get_object
method. Here's an example from the documentation on how to use the S3Resource
to interact with S3: python from dagster import job, op, Definitions from dagster_aws.s3 import S3Resource @op def example_s3_op(s3: S3Resource): s3_client = s3.get_client() return s3_client.list_objects_v2( Bucket='my-bucket', Prefix='some-key' ) @job def example_job(): example_s3_op() defs = Definitions( jobs=[example_job], resources={'s3': S3Resource(region_name='us-west-1')} )
In this example, the example_s3_op
operation uses the S3Resource
to get a boto3 client and then calls list_objects_v2
on that client. If you want to retrieve an object, you would use the get_object
method of the boto3 client, like so: python s3_client = s3.get_client() response = s3_client.get_object(Bucket='my-bucket', Key='my-key')
Remember to replace 'my-bucket'
and 'my-key'
with the actual bucket name and object key you are trying to retrieve. The response
will contain the object data, which you can access using response['Body'].read()
if you want to read the content of the file. If you encounter any issues or errors, ensure that your S3Resource
is correctly configured with the necessary credentials and permissions to access the S3 bucket and objects.