https://dagster.io/ logo
#community-showcase
Title
# community-showcase
z

Zach

05/04/2022, 2:18 PM
I made a little SSMSource config schema Field Shape that I've found super useful as I store a lot of parameters in AWS Systems Manager Parameter store that get updated by other CI/CD processes but need to get pulled in to run configs at runtime. I thought other people might find it useful so I put it in a gist to share: https://gist.github.com/zyd14/1280e424e32d81e5bfb74eb81913706a it basically allows you to do this:
Copy code
@op(
    config_schema = {'param': Field(SSMSource)}
)
def do_something(context):
    ssm_param_value = context.op_config['param']
    ...

ops:
  do_something:
    param:
      ssm_config:
        param_name: some_param_in_ssm
        encrypted: True
OR
ops:
  do_something:
    param: some_scalar_value
and then some_param_in_ssm gets replaced with the value found in AWS Systems Manager Parameter Store at runtime. my only hangup with it currently is that it needs a boto3 connection outside of when you have access to dagster context resources, so it's a bit harder to mock that part out if you're testing run configs that try to retrieve values from SSM - but generally you can pass scalar strings into you run_config when testing instead of using the ssm_config selector to avoid the boto3 connection.
❤️ 1
s

Stephen Bailey

05/04/2022, 5:41 PM
this is great. i though about doing something like this but ended up just writing an
get_aws_ssm_secret
helper function, and configuring resources during the collection step, rather than allowing them to be configured via config
it would be great to have an out of the box pattern for this type of thing!