Hello Dagster Community, It’s been a very long time since I ask, so maybe things changed. What’s the current

best

way to configure

AWS Secrets

in Dagster

Resources

What I want is to store all my credentials in

AWS Secrets

, and let

Dagster

create the needed

Resources

with the right

AWS Secret

when the job run. Right now, in the configuration I have, Dagster will load all Secrets as environment variables directly when It’s starts. When I work on something that doesn’t requires any AWS Secrets, it’s sad that my configuration still need a valid

AWS_PROFILE

configured in order to load the project and start dagster. How do you guys deal with this issue? Here’s what my configuration look like (For

Snowflake

) : I load all credentials to environment variables using

boto3

manually

# Load snowflake credentials in environment variable
# before creating the ressources.
load_snowflake_credentials(DEPLOYMENT_NAME)

I then configure the needed Snowflake Config

SHARED_SNOWFLAKE_CONF

SHARED_SNOWFLAKE_CONF = {
    "account": os.getenv("SNOWFLAKE_ACCOUNT", ""),
    "user": os.getenv("SNOWFLAKE_USER", ""),
    "password": os.getenv("SNOWFLAKE_PASSWORD", ""),
    "warehouse": os.getenv("SNOWFLAKE_WAREHOUSE", ""),
    "database": os.getenv("SNOWFLAKE_DATABASE", ""),
    "role": os.getenv("SNOWFLAKE_ROLE", ""),
}

And then attach that config to

snowflake_io_manager

RESOURCES_LOCAL = {
  "warehouse_io_manager": snowflake_io_manager.configured(
        dict(**SHARED_SNOWFLAKE_CONF)
    ),
}

Is there a better way to do this that would allow me to still give my

asset

a _`required_resource_keys`,_ but build it at run time instead?

If you were okay with creating your own resource you could probably just take the snowflake IO manager class and wrap it in your own resource, where all the values get set during initialization instead of using

.configured()

at the global level. Something like

from dagster import resource
from dagster_snowflake import SnowflakeIOManager

@resource
def deferred_setup_snowflake_io_manager():
  SHARED_SNOWFLAKE_CONF = {
    "account": os.getenv("SNOWFLAKE_ACCOUNT", ""),
    "user": os.getenv("SNOWFLAKE_USER", ""),
    "password": os.getenv("SNOWFLAKE_PASSWORD", ""),
    "warehouse": os.getenv("SNOWFLAKE_WAREHOUSE", ""),
    "database": os.getenv("SNOWFLAKE_DATABASE", ""),
    "role": os.getenv("SNOWFLAKE_ROLE", ""),
  }
  return SnowflakeIOManager(**SHARED_SNOWFLAKE_CONF)

Then the environment variables only get accessed when the snowflake IO manager gets used, instead of when your code is loaded

Hey Jacob, I’m not sure I fully understand your issue but I know there have been changes in this area recently. Have you looked at our most recent docs on secrets management? https://docs.dagster.io/guides/dagster/using-environment-variables-and-secrets#handling-secrets

@sean I am struggling with this same issue. I have multiple resources that require configuration secrets that are stored in AWS Secrets Manager. This feels like a common need, but I cannot find any elegant solutions within Dagster. The main roadblock is as @Jacob Marcil said, that any attempt to load AWS Secrets into environment variables before configuring resources will be triggered in every context, from dagster dev to loading code. We would only like the “AWS Secret -> environment variable” code to run in the context of a job about to run. Is there a simple way to accomplish this, or is this the wrong solution?