My database credentials are stored in a

secretsmanager

secret, as a JSON secret something like this: secretsmanager

{
  "dbname": "...",
  "engine": "mysql",
  "host": "...",
  "password": "...",
  "port": 3306,
  "username": "..."
}

But the storage params are something like this: dagster.yml

storage:
  mysql:
    mysql_db:
      username:
        env: DAGSTER_MYSQL_USERNAME
      ...

So, in my docker-compose I mount the secrets like this and have a script that reads them: docker-compose.yml

secrets:
  db_credentials:
    external: true
    name: ${DB_CREDENTIALS_SECRET_ARN}
    x-aws-keys:
      - "*"

env-wrap.sh

...
[ -f /run/secrets/db_credentials/username ] && export DAGSTER_MYSQL_USERNAME=$(cat /run/secrets/db_credentials/username)
...

This is fine for the daemon, dagit and the user_code services because I can specify my own

CMD

but in the case of launched tasks, it overrides my

CMD

and

ENTRYPOINT

that I set to make this work, giving me the following error:

dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_MYSQL_USERNAME" which is not set. In order for this execution to succeed it must be set in this environment.

Any ideas on how to make secretsmanager JSON export the variables correctly for tasks launched by dagster? Thanks!

We're using our own CloudFormation-based setup instead of the Docker Compose, that takes the ARN. I wonder if you could move the functionality of that script to whatever runs the Docker Compose.

Hmm, I don’t think that works for me because I don’t want the credentials built into the image

We do set some secrets using run launcher config also using ARNs, that may work for launched tasks.

Yea that might work to get the credentials in the image like I have it, but I’m trying to export them to the env as variables using a script. But my script isn’t running

That does expose them as env vars, I think.

Hmm, does it support parsing the secret as JSON?

I reference the specific key in the JSON.

Didn’t know you could do that

Yeah, looking in our task definition our database secrets (not the ones in the

runLauncher

config) are just resolved at build time and baked in. Could maybe improve on that, if ECS supports it.

Hi Mitchell - not sure if you got sorted out or not, but you can provide secrets in your run launcher config to mount those secrets as environment variables in launched tasks: https://docs.dagster.io/deployment/guides/aws#secrets-management-in-ecs You can either provide the specific ARNs you want to include or you can provide a tag to mount any ARNs tagged with that key.

Thanks Jordan. Yeah, I did figure it out, something like this:

run_launcher:
  module: dagster_aws.ecs
  class: "EcsRunLauncher"
  config:
    secrets:
      - name: DAGSTER_MYSQL_USERNAME
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:username::"
      - name: DAGSTER_MYSQL_PASSWORD
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:password::"
      - name: DAGSTER_MYSQL_HOSTNAME
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:host::"
      - name: DAGSTER_MYSQL_DB
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:dbname::"

Then I did this in my dockerfile:

RUN sed -i "s|\$DB_CREDENTIALS_SECRET_ARN|$DB_CREDENTIALS_SECRET_ARN|g" $DAGSTER_HOME/dagster.yaml