https://dagster.io/ logo
#dagster-ecs
Title
# dagster-ecs
m

Mitchell Hynes

01/29/2023, 2:41 PM
My database credentials are stored in a
secretsmanager
secret, as a JSON secret something like this: secretsmanager
Copy code
{
  "dbname": "...",
  "engine": "mysql",
  "host": "...",
  "password": "...",
  "port": 3306,
  "username": "..."
}
But the storage params are something like this: dagster.yml
Copy code
storage:
  mysql:
    mysql_db:
      username:
        env: DAGSTER_MYSQL_USERNAME
      ...
So, in my docker-compose I mount the secrets like this and have a script that reads them: docker-compose.yml
Copy code
secrets:
  db_credentials:
    external: true
    name: ${DB_CREDENTIALS_SECRET_ARN}
    x-aws-keys:
      - "*"
env-wrap.sh
Copy code
...
[ -f /run/secrets/db_credentials/username ] && export DAGSTER_MYSQL_USERNAME=$(cat /run/secrets/db_credentials/username)
...
This is fine for the daemon, dagit and the user_code services because I can specify my own
CMD
but in the case of launched tasks, it overrides my
CMD
and
ENTRYPOINT
that I set to make this work, giving me the following error:
Copy code
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_MYSQL_USERNAME" which is not set. In order for this execution to succeed it must be set in this environment.
Any ideas on how to make secretsmanager JSON export the variables correctly for tasks launched by dagster? Thanks!
It would be awesome if I could just supply
mysql_db:
with a secretsmanager ARN, but that feels like it’s out of the scope of responsibilities for
dagster_mysql
.
c

Chris Zubak-Skees

01/29/2023, 5:39 PM
We're using our own CloudFormation-based setup instead of the Docker Compose, that takes the ARN. I wonder if you could move the functionality of that script to whatever runs the Docker Compose.
Unpacking the secrets at deploy time instead of run time.
m

Mitchell Hynes

01/29/2023, 5:43 PM
Hmm, I don’t think that works for me because I don’t want the credentials built into the image
I also don’t want them in the task definition as env variables
👍 1
c

Chris Zubak-Skees

01/29/2023, 5:48 PM
We do set some secrets using run launcher config also using ARNs, that may work for launched tasks.
e.g.
Copy code
run_launcher:
  config:
    secrets:
      - name: ...
        valueFrom: "arn:aws:secretsmanager:
m

Mitchell Hynes

01/29/2023, 5:49 PM
Yea that might work to get the credentials in the image like I have it, but I’m trying to export them to the env as variables using a script. But my script isn’t running
c

Chris Zubak-Skees

01/29/2023, 5:49 PM
That does expose them as env vars, I think.
Could be wrong.
m

Mitchell Hynes

01/29/2023, 5:50 PM
Hmm, does it support parsing the secret as JSON?
c

Chris Zubak-Skees

01/29/2023, 5:50 PM
I reference the specific key in the JSON.
In the ARN.
👍 1
m

Mitchell Hynes

01/29/2023, 5:51 PM
Didn’t know you could do that
c

Chris Zubak-Skees

01/29/2023, 5:52 PM
Yeah, looking in our task definition our database secrets (not the ones in the
runLauncher
config) are just resolved at build time and baked in. Could maybe improve on that, if ECS supports it.
j

jordan

01/30/2023, 6:26 PM
Hi Mitchell - not sure if you got sorted out or not, but you can provide secrets in your run launcher config to mount those secrets as environment variables in launched tasks: https://docs.dagster.io/deployment/guides/aws#secrets-management-in-ecs You can either provide the specific ARNs you want to include or you can provide a tag to mount any ARNs tagged with that key.
m

Mitchell Hynes

01/30/2023, 6:34 PM
Thanks Jordan. Yeah, I did figure it out, something like this:
Copy code
run_launcher:
  module: dagster_aws.ecs
  class: "EcsRunLauncher"
  config:
    secrets:
      - name: DAGSTER_MYSQL_USERNAME
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:username::"
      - name: DAGSTER_MYSQL_PASSWORD
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:password::"
      - name: DAGSTER_MYSQL_HOSTNAME
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:host::"
      - name: DAGSTER_MYSQL_DB
        valueFrom: "$DB_CREDENTIALS_SECRET_ARN:dbname::"
Then I did this in my dockerfile:
Copy code
RUN sed -i "s|\$DB_CREDENTIALS_SECRET_ARN|$DB_CREDENTIALS_SECRET_ARN|g" $DAGSTER_HOME/dagster.yaml
🙌 1
7 Views