https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jacob Marcil

08/14/2023, 9:40 PM
Hello Dagster Community, It’s been a very long time since I ask, so maybe things changed. What’s the current
best
way to configure
AWS Secrets
in Dagster
Resources
What I want is to store all my credentials in
AWS Secrets
, and let
Dagster
create the needed
Resources
with the right
AWS Secret
when the job run. Right now, in the configuration I have, Dagster will load all Secrets as environment variables directly when It’s starts. When I work on something that doesn’t requires any AWS Secrets, it’s sad that my configuration still need a valid
AWS_PROFILE
configured in order to load the project and start dagster. How do you guys deal with this issue? Here’s what my configuration look like (For
Snowflake
) : I load all credentials to environment variables using
boto3
manually
Copy code
# Load snowflake credentials in environment variable
# before creating the ressources.
load_snowflake_credentials(DEPLOYMENT_NAME)
I then configure the needed Snowflake Config
SHARED_SNOWFLAKE_CONF
Copy code
SHARED_SNOWFLAKE_CONF = {
    "account": os.getenv("SNOWFLAKE_ACCOUNT", ""),
    "user": os.getenv("SNOWFLAKE_USER", ""),
    "password": os.getenv("SNOWFLAKE_PASSWORD", ""),
    "warehouse": os.getenv("SNOWFLAKE_WAREHOUSE", ""),
    "database": os.getenv("SNOWFLAKE_DATABASE", ""),
    "role": os.getenv("SNOWFLAKE_ROLE", ""),
}
And then attach that config to
snowflake_io_manager
Copy code
RESOURCES_LOCAL = {
  "warehouse_io_manager": snowflake_io_manager.configured(
        dict(**SHARED_SNOWFLAKE_CONF)
    ),
}
Is there a better way to do this that would allow me to still give my
asset
a _`required_resource_keys`,_ but build it at run time instead?
🤖 1
z

Zach

08/15/2023, 3:49 PM
If you were okay with creating your own resource you could probably just take the snowflake IO manager class and wrap it in your own resource, where all the values get set during initialization instead of using
.configured()
at the global level. Something like
Copy code
from dagster import resource
from dagster_snowflake import SnowflakeIOManager

@resource
def deferred_setup_snowflake_io_manager():
  SHARED_SNOWFLAKE_CONF = {
    "account": os.getenv("SNOWFLAKE_ACCOUNT", ""),
    "user": os.getenv("SNOWFLAKE_USER", ""),
    "password": os.getenv("SNOWFLAKE_PASSWORD", ""),
    "warehouse": os.getenv("SNOWFLAKE_WAREHOUSE", ""),
    "database": os.getenv("SNOWFLAKE_DATABASE", ""),
    "role": os.getenv("SNOWFLAKE_ROLE", ""),
  }
  return SnowflakeIOManager(**SHARED_SNOWFLAKE_CONF)
Then the environment variables only get accessed when the snowflake IO manager gets used, instead of when your code is loaded
🌈 1
s

sean

08/15/2023, 7:00 PM
Hey Jacob, I’m not sure I fully understand your issue but I know there have been changes in this area recently. Have you looked at our most recent docs on secrets management? https://docs.dagster.io/guides/dagster/using-environment-variables-and-secrets#handling-secrets
n

Nathan Skone

10/17/2023, 7:38 PM
@sean I am struggling with this same issue. I have multiple resources that require configuration secrets that are stored in AWS Secrets Manager. This feels like a common need, but I cannot find any elegant solutions within Dagster. The main roadblock is as @Jacob Marcil said, that any attempt to load AWS Secrets into environment variables before configuring resources will be triggered in every context, from dagster dev to loading code. We would only like the “AWS Secret -> environment variable” code to run in the context of a job about to run. Is there a simple way to accomplish this, or is this the wrong solution?
13 Views