Hey all, I have a Docker/Dagster question—right now in my Dagster pipeline I'm using the

dagster-aws

library to read/write files to S3.  It works locally, however, when running within Docker, it can't find my credentials. The specific error is

botocore.exceptions.NoCredentialsError: Unable to locate credentials

and seems to be coming from

dagster_aws/s3/compute_log_manager.py

when boto3 tries to instantiate the client. I volume mounted my 

~/.aws

 file (containing creds/config) into the Docker containers at 

/root

 using

docker-compose

but Dagster/boto3 still isn't able to find the credentials.  Does anyone know how to resolve this/where the credentials should be stores so boto3 can find them? (note for actual deployment we plan to use IAM but for testing Docker locally I need to figure out how to pass the creds).  Thanks again!

could try environment variables

Are you using a single docker container or running a multi-container setup with docker-compose

multi-container setup with docker-compose

It does, it just uses botocore so it’ll look for env vars to provide the credentials

Oh awesome. That would be ideal then. This was the line where I think the client gets created right (

python_modules/libraries/dagster-aws/dagster_aws/s3/compute_log_manager.py

line 63)?

self._s3_session = boto3.resource(
            "s3", use_ssl=use_ssl, verify=_verify, endpoint_url=endpoint_url
        ).meta.client

If you don’t have the var’s set by default you can do:

AWS_ACCESS_KEY_ID="$(aws configure get aws_access_key_id)"
AWS_SECRET_ACCESS_KEY="$(aws configure get aws_secret_access_key)"

in a build.sh script. Just need to expose them in the docker-compose for each service, I don’t think I even reference them in the actual dockerfiles

Oh awesome thanks!

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials boto has a credential lookup chain where it will look in these places in this order. So https://github.com/dagster-io/dagster/blob/0d85183fda0654dea76638e73dd4fc88fb825b1[…]les/libraries/dagster-aws/dagster_aws/s3/compute_log_manager.py will start at 3. on the list (envvars)

Oh awesome thanks! That was the piece of info I was missing. Really appreciate it all!