Anyone know how to upload a file to s3 using resources?

from dagster_aws.s3 import s3_resource



@op(required_resource_keys={'s3'})
def load_json_to_s3(context, json_game_data_list):
    for json in json_game_data_list:
        context.resources.s3.put_object(
        body=json,
        Bucket='nhl-analytics',
        Prefix='test'
        )

Could take a looks at https://docs.dagster.io/deployment/guides/aws#using-s3-for-io-management A common way is with

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

yeah I usually just configure aws creds as the standard boto AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY environment variables and the s3 resource works. it's just a boto3 s3 client so you can configure it using a credentials profile too

where do you place the keys?

In the env wherever your Op is running. That would depend on how you have Dagster deployed

got it, so I can store them as environment variables

Yep

how would I then specify a profile?

Different options for configuring boto: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables

say I have to aws profiles set in my environment