Mitchell Hynes
01/29/2023, 2:41 PMsecretsmanager
secret, as a JSON secret something like this:
secretsmanager
{
"dbname": "...",
"engine": "mysql",
"host": "...",
"password": "...",
"port": 3306,
"username": "..."
}
But the storage params are something like this:
dagster.yml
storage:
mysql:
mysql_db:
username:
env: DAGSTER_MYSQL_USERNAME
...
So, in my docker-compose I mount the secrets like this and have a script that reads them:
docker-compose.yml
secrets:
db_credentials:
external: true
name: ${DB_CREDENTIALS_SECRET_ARN}
x-aws-keys:
- "*"
env-wrap.sh
...
[ -f /run/secrets/db_credentials/username ] && export DAGSTER_MYSQL_USERNAME=$(cat /run/secrets/db_credentials/username)
...
This is fine for the daemon, dagit and the user_code services because I can specify my own CMD
but in the case of launched tasks, it overrides my CMD
and ENTRYPOINT
that I set to make this work, giving me the following error:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_MYSQL_USERNAME" which is not set. In order for this execution to succeed it must be set in this environment.
Any ideas on how to make secretsmanager JSON export the variables correctly for tasks launched by dagster? Thanks!mysql_db:
with a secretsmanager ARN, but that feels like it’s out of the scope of responsibilities for dagster_mysql
.Chris Zubak-Skees
01/29/2023, 5:39 PMMitchell Hynes
01/29/2023, 5:43 PMChris Zubak-Skees
01/29/2023, 5:48 PMrun_launcher:
config:
secrets:
- name: ...
valueFrom: "arn:aws:secretsmanager:
Mitchell Hynes
01/29/2023, 5:49 PMChris Zubak-Skees
01/29/2023, 5:49 PMMitchell Hynes
01/29/2023, 5:50 PMChris Zubak-Skees
01/29/2023, 5:50 PMMitchell Hynes
01/29/2023, 5:51 PMChris Zubak-Skees
01/29/2023, 5:52 PMrunLauncher
config) are just resolved at build time and baked in. Could maybe improve on that, if ECS supports it.jordan
01/30/2023, 6:26 PMMitchell Hynes
01/30/2023, 6:34 PMrun_launcher:
module: dagster_aws.ecs
class: "EcsRunLauncher"
config:
secrets:
- name: DAGSTER_MYSQL_USERNAME
valueFrom: "$DB_CREDENTIALS_SECRET_ARN:username::"
- name: DAGSTER_MYSQL_PASSWORD
valueFrom: "$DB_CREDENTIALS_SECRET_ARN:password::"
- name: DAGSTER_MYSQL_HOSTNAME
valueFrom: "$DB_CREDENTIALS_SECRET_ARN:host::"
- name: DAGSTER_MYSQL_DB
valueFrom: "$DB_CREDENTIALS_SECRET_ARN:dbname::"
Then I did this in my dockerfile:
RUN sed -i "s|\$DB_CREDENTIALS_SECRET_ARN|$DB_CREDENTIALS_SECRET_ARN|g" $DAGSTER_HOME/dagster.yaml