Hello! For handling AWS secrets, the docs state that “By default, Dagster will fetch any Secrets Manager secrets tagged with the

dagster

key and set them as environment variables.” Two questions: • To confirm, this is referring to a tag with key = “dagster”, right? If so, can the value of that tag (whose key is

dagster

) be anything? • At what point does the secret actually get stored as an environment variable? I know we need to give the task IAM role the relevant secrets access, but I’d love to learn when & how the secret actually gets carried over into the relevant container(s).

• confirmed! • The values are pulled from the secret and added as environment variables by ECS when the container is spun up (before any python code runs). More information about this process in the ECS docs here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html - dagster is setting the "secrets" key described in those. docs, e.g.

{
  "containerDefinitions": [
    {
      "secrets": [
        {
          "name": "environment_variable_name",
          "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
        }
      ]
    }
  ]
}

Perfect, thank you!

dagster doesn't actually pick the environment variable names, ECS does - ECS picks all the key-value pairs in the secret, and I don't actually know how ECS breaks ties if the same key is specified more than once

ah okay, but since ECS does, it’ll just pick all key-value pairs in every secret tagged with

dagster

. that helps ! For context, I have a secret with that tag, but none of my containers are picking it up - even after restarting the ECS service. are there any common errors that you’ve seen for configuring secrets ?

You could try specifying the secret by ARN first and see if that works

oooh thank you, I’ll try that! When you say “if the secret was specified”, are you referring to including it in the

environment

block ?

It should look like that example I posted earlier:

{
  "containerDefinitions": [
    {
      "secrets": [
        {
          "name": "environment_variable_name",
          "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
        }
      ]
    }
  ]
}

ahhhhh

yeah, the ECS task that is created for each run

Cool, thanks for confirming! i’m going to try hard-coding the secret ARN. In the meantime, when you suggested “checking the task definition to see if the secret was specified can help rule out problems”, does that mean we need to include anything for the secret in the task definition itself?

You don't need to include anything, I was just suggesting that to confirm that it made it onto the task definition to rule out a problem there

Oh I see! I do see it in the task command (

["dagster","api","execute_run",...]

) under

run_launcher_data

, but it does’t flow through as an environment variable. Does that mean Dagster knows what to pull but can’t because of IAM issues?

If there were IAM issues I would expect there to be an exception, it seems like it's not finding any secrets with that tag. Are you sure your boto3 client is using the right region? You could also try passing in a specific secret ARN first using

secrets

instead of

secrets_tag

and see if that is working and verify that it is looking for secrets in the right place, then try the tags thing

Yeah, passing in the hard-coded secret isn’t working either, and confirmed that the region is correct. I might revisit this later, but for now I’ll just pass in the secrets directly in the task definition. Thanks for the help!