https://dagster.io/ logo
#deployment-ecs
Title
# deployment-ecs
s

Soonho Kwon

06/15/2023, 4:09 AM
Hello! For handling AWS secrets, the docs state that “By default, Dagster will fetch any Secrets Manager secrets tagged with the
dagster
key and set them as environment variables.” Two questions: • To confirm, this is referring to a tag with key = “dagster”, right? If so, can the value of that tag (whose key is
dagster
) be anything? • At what point does the secret actually get stored as an environment variable? I know we need to give the task IAM role the relevant secrets access, but I’d love to learn when & how the secret actually gets carried over into the relevant container(s).
d

daniel

06/15/2023, 1:55 PM
• confirmed! • The values are pulled from the secret and added as environment variables by ECS when the container is spun up (before any python code runs). More information about this process in the ECS docs here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html - dagster is setting the "secrets" key described in those. docs, e.g.
Copy code
{
  "containerDefinitions": [
    {
      "secrets": [
        {
          "name": "environment_variable_name",
          "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
        }
      ]
    }
  ]
}
s

Soonho Kwon

06/15/2023, 3:09 PM
Perfect, thank you!
Actually — @daniel a couple of more questions: • How does dagster pick the
environment_variable_name
? is that the name of the secret itself (in which case is it assuming plain-text secrets vs. key/val?), or the key in the secret key/val pair? • If the latter: ◦ if a secret has multiple key/val pairs will Dagster pull in all of them? ◦ how would Dagster handle it if two different secrets have the same key but different values?
d

daniel

06/15/2023, 6:27 PM
dagster doesn't actually pick the environment variable names, ECS does - ECS picks all the key-value pairs in the secret, and I don't actually know how ECS breaks ties if the same key is specified more than once
s

Soonho Kwon

06/15/2023, 7:01 PM
ah okay, but since ECS does, it’ll just pick all key-value pairs in every secret tagged with
dagster
. that helps ! For context, I have a secret with that tag, but none of my containers are picking it up - even after restarting the ECS service. are there any common errors that you’ve seen for configuring secrets ?
d

daniel

06/15/2023, 7:02 PM
You could try specifying the secret by ARN first and see if that works
making sure its in the correct region and AWS account and that your app has the right permissions to access secrets is another gotcha
checking the task definition to see if the secret was specified can help rule out problems
s

Soonho Kwon

06/15/2023, 7:04 PM
oooh thank you, I’ll try that! When you say “if the secret was specified”, are you referring to including it in the
environment
block ?
d

daniel

06/15/2023, 7:04 PM
It should look like that example I posted earlier:
Copy code
{
  "containerDefinitions": [
    {
      "secrets": [
        {
          "name": "environment_variable_name",
          "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
        }
      ]
    }
  ]
}
Or wait, sorry, not exactly that, one moment
I'm sorry, I think i misremembered the details here. if you're using the option where it's tagged with
dagster
, it uses the name of the secret as the environment variable name, and expects the secret to have a single value
s

Soonho Kwon

06/15/2023, 7:12 PM
ahhhhh
okay, I will try that. when you said the secret gets added “when the container is spun up” , are you referring to just the container triggered by the runner?
d

daniel

06/15/2023, 7:15 PM
yeah, the ECS task that is created for each run
s

Soonho Kwon

06/15/2023, 8:30 PM
Cool, thanks for confirming! i’m going to try hard-coding the secret ARN. In the meantime, when you suggested “checking the task definition to see if the secret was specified can help rule out problems”, does that mean we need to include anything for the secret in the task definition itself?
d

daniel

06/15/2023, 9:01 PM
You don't need to include anything, I was just suggesting that to confirm that it made it onto the task definition to rule out a problem there
s

Soonho Kwon

06/15/2023, 11:27 PM
Oh I see! I do see it in the task command (
["dagster","api","execute_run",...]
) under
run_launcher_data
, but it does’t flow through as an environment variable. Does that mean Dagster knows what to pull but can’t because of IAM issues?
d

daniel

06/16/2023, 1:58 PM
If there were IAM issues I would expect there to be an exception, it seems like it's not finding any secrets with that tag. Are you sure your boto3 client is using the right region? You could also try passing in a specific secret ARN first using
secrets
instead of
secrets_tag
and see if that is working and verify that it is looking for secrets in the right place, then try the tags thing
s

Soonho Kwon

06/16/2023, 7:43 PM
Yeah, passing in the hard-coded secret isn’t working either, and confirmed that the region is correct. I might revisit this later, but for now I’ll just pass in the secrets directly in the task definition. Thanks for the help!
8 Views