Jason Vorenkamp07/01/2022, 8:22 PM
I was hoping there would be some way to pass in this configuration but it looks like additional arguments are not allowed: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-aws/dagster_aws/emr/pyspark_step_launcher.py#L274 Is there some other way to add configuration to have the upload specify server side encryption?
$ aws s3 cp test.txt <s3://staging-bucket/test.txt> upload failed: ./test.txt to <s3://staging-bucket/test.txt> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied $ aws s3 cp test.txt <s3://staging-bucket/test.txt> --sse AES256 upload: ./test.txt to <s3://staging-bucket/test.txt>
owen07/01/2022, 8:50 PM
method. Also, if you're interested in contributing a change for this behavior, we'd be happy to review that for you!
Jason Vorenkamp07/01/2022, 9:04 PM