https://dagster.io/ logo
#ask-community
Title
# ask-community
g

geoHeil

05/15/2022, 3:24 PM
I am facing https://stackoverflow.com/questions/67093837/how-to-load-file-from-custom-hosted-minio-s3-bucket-into-pandas-using-s3-url-for when tying to use the existing S3 resource of dagster with pandas https://docs.dagster.io/_apidocs/libraries/dagster-aws#s3 is there any way I do not need to re-implement the resource myself to be able to access the properties as required? I could not find a generic S3 resource and are not sure how to get AWS_ACCESS_KEY_ID / secret out of the initialised (existing) dagster S3 resource (to be able to pass them to pandas)? I think https://stackoverflow.com/questions/72248954/boto3-s3-initialized-session-return-credentials is a more generic formulation of this request (with a nice suggestion). However, this fails in the case of dagster as the
_credentials
attribute is
None
for the readily initialized
s3
resource. I do not know why it is not initialized.
🤖 1
j

jamie

05/16/2022, 6:38 PM
Hey @geoHeil Based on the Stack overflow post about the
get_credentials
function, I think that only applies to a generic aws session (ie the return of
boto3.Session()
) in the S3 resource we are returning an instance of an S3 client. The docs for the S3 client (here) don't indicate any way to get the credentials from the S3 client. One option would be to write a simple AWS session resource that can return credentials and add that as a required resource to your op
g

geoHeil

05/16/2022, 9:20 PM
No. You are referring to the first part of the answer. When you look at the 2nd part - the S3 client then it should work.
But for me the _credentials field is not populated
j

jamie

05/16/2022, 9:23 PM
ah i see - didn't scroll down enough. i'll dig into our code a bit and see if i can find out what's going on. from what i can see right now we just instantiate an S3 client in a normal way so i'm not sure yet why
_credentials
wouldn't be populated. i'll get back to you
The following works for me
Copy code
@op(
    required_resource_keys={"s3_resource"}
)
def get_s3_creds(context):
    s3 = context.resources.s3_resource
    creds = s3._request_signer._credentials
    <http://context.log.info|context.log.info>(f"CREDENTIALS {creds.secret_key} {creds.token} {creds.access_key}")


@job(
    resource_defs={"s3_resource": s3_resource}
)
def s3_job():
    get_s3_creds()
the
_credentials
attr is under
_request_signer
g

geoHeil

05/18/2022, 8:00 AM
great! this works nicely for the credentials.
1