https://dagster.io/ logo
#ask-ai
Title
# ask-ai
b

Binoy Shah

02/22/2024, 8:55 PM
@Scout I am using
dagster-polars
library for PolarsParquetIOManager. I am struggling to set up correct S3 Access. Our S3 credentials are not in their standard format and i need to copy it from some other place, How do I provide the PolarsParquetIOManager correct credentials
s

Scout

02/22/2024, 8:55 PM
To provide the
PolarsParquetIOManager
with the correct S3 credentials, you would typically use the configuration options available for S3 resources in Dagster. The
dagster-aws
library provides a resource called
s3_file_manager
, which can be configured with S3 credentials. The documentation provides an example of how to configure S3 resources with various options, including credentials. Here's a snippet from the documentation that shows the configuration schema for S3 resources:
Copy code
yaml resources:   s3:     config:       region_name: "us-west-1"       use_unsigned_session: false       endpoint_url: "<http://localhost>"       profile_name: "dev"       use_ssl: true       verify: None       aws_access_key_id: None       aws_secret_access_key: None       aws_session_token: None
In this schema, you can provide the
aws_access_key_id
,
aws_secret_access_key
, and
aws_session_token
directly if you have them. If your credentials are not in the standard AWS credentials file format and you need to copy them from another place, you can set these values in your Dagster resource configuration, either directly in your code or via environment variables. If you're using environment variables, you can set them in your system or pass them through your deployment configuration, depending on how you run your Dagster pipelines. For example, you could set the following environment variables:
Copy code
sh export AWS_ACCESS_KEY_ID=your_access_key_id export AWS_SECRET_ACCESS_KEY=your_secret_access_key export AWS_SESSION_TOKEN=your_session_token
Then, in your Dagster resource configuration, you can reference these environment variables:
Copy code
yaml resources:   s3:     config:       aws_access_key_id:         env: AWS_ACCESS_KEY_ID       aws_secret_access_key:         env: AWS_SECRET_ACCESS_KEY       aws_session_token:         env: AWS_SESSION_TOKEN
Please note that the
PolarsParquetIOManager
might have its own specific way of handling credentials if it's a custom IO manager from the
dagster-polars
library. The above instructions are based on the standard way of configuring S3 resources in Dagster. If
dagster-polars
requires a different method, you should refer to the specific documentation for that library or reach out to the maintainers for guidance on how to provide custom credentials.
2 Views