@Scout I am using

dagster-polars

library for PolarsParquetIOManager. I am struggling to set up correct S3 Access. Our S3 credentials are not in their standard format and i need to copy it from some other place, How do I provide the PolarsParquetIOManager correct credentials

To provide the

PolarsParquetIOManager

with the correct S3 credentials, you would typically use the configuration options available for S3 resources in Dagster. The

dagster-aws

library provides a resource called

s3_file_manager

, which can be configured with S3 credentials. The documentation provides an example of how to configure S3 resources with various options, including credentials. Here's a snippet from the documentation that shows the configuration schema for S3 resources:

yaml resources:   s3:     config:       region_name: "us-west-1"       use_unsigned_session: false       endpoint_url: "<http://localhost>"       profile_name: "dev"       use_ssl: true       verify: None       aws_access_key_id: None       aws_secret_access_key: None       aws_session_token: None

In this schema, you can provide the

aws_access_key_id

,

aws_secret_access_key

, and

aws_session_token

directly if you have them. If your credentials are not in the standard AWS credentials file format and you need to copy them from another place, you can set these values in your Dagster resource configuration, either directly in your code or via environment variables. If you're using environment variables, you can set them in your system or pass them through your deployment configuration, depending on how you run your Dagster pipelines. For example, you could set the following environment variables:

sh export AWS_ACCESS_KEY_ID=your_access_key_id export AWS_SECRET_ACCESS_KEY=your_secret_access_key export AWS_SESSION_TOKEN=your_session_token

Then, in your Dagster resource configuration, you can reference these environment variables:

yaml resources:   s3:     config:       aws_access_key_id:         env: AWS_ACCESS_KEY_ID       aws_secret_access_key:         env: AWS_SECRET_ACCESS_KEY       aws_session_token:         env: AWS_SESSION_TOKEN

Please note that the

PolarsParquetIOManager

might have its own specific way of handling credentials if it's a custom IO manager from the

dagster-polars

library. The above instructions are based on the standard way of configuring S3 resources in Dagster. If

dagster-polars

requires a different method, you should refer to the specific documentation for that library or reach out to the maintainers for guidance on how to provide custom credentials.