https://dagster.io/ logo
Join Slack
Powered by
I am facing <https://stackoverflow.com/questions/6...
# ask-community
g
I am facing https://stackoverflow.com/questions/67093837/how-to-load-file-from-custom-hosted-minio-s3-bucket-into-pandas-using-s3-url-for when tying to use the existing S3 resource of dagster with pandas https://docs.dagster.io/_apidocs/libraries/dagster-aws#s3 is there any way I do not need to re-implement the resource myself to be able to access the properties as required? I could not find a generic S3 resource and are not sure how to get AWS_ACCESS_KEY_ID / secret out of the initialised (existing) dagster S3 resource (to be able to pass them to pandas)? I think https://stackoverflow.com/questions/72248954/boto3-s3-initialized-session-return-credentials is a more generic formulation of this request (with a nice suggestion). However, this fails in the case of dagster as the 
_credentials
attribute is 
None
for the readily initialized 
s3
resource. I do not know why it is not initialized.
🤖 1
j
Hey @geoHeil Based on the Stack overflow post about the 
get_credentials
function, I think that only applies to a generic aws session (ie the return of 
boto3.Session()
) in the S3 resource we are returning an instance of an S3 client. The docs for the S3 client (here) don't indicate any way to get the credentials from the S3 client. One option would be to write a simple AWS session resource that can return credentials and add that as a required resource to your op
g
No. You are referring to the first part of the answer. When you look at the 2nd part - the S3 client then it should work.
But for me the _credentials field is not populated
j
ah i see - didn't scroll down enough. i'll dig into our code a bit and see if i can find out what's going on. from what i can see right now we just instantiate an S3 client in a normal way so i'm not sure yet why 
_credentials
wouldn't be populated. i'll get back to you
The following works for me
Copy code
@op(
    required_resource_keys={"s3_resource"}
)
def get_s3_creds(context):
    s3 = context.resources.s3_resource
    creds = s3._request_signer._credentials
    <http://context.log.info|context.log.info>(f"CREDENTIALS {creds.secret_key} {creds.token} {creds.access_key}")


@job(
    resource_defs={"s3_resource": s3_resource}
)
def s3_job():
    get_s3_creds()
the 
_credentials
attr is under 
_request_signer
g
great! this works nicely for the credentials.
✅ 1
Open in Slack
PreviousNext
Powered by