Hello! I am attempting to get Dagster setup in ECS...
# deployment-ecs
n
Hello! I am attempting to get Dagster setup in ECS using docker compose. I am having an issue where if I setup secrets to be retrieved from AWS Secret Manager in the dagster.yaml file and then calling those as EnvVar for Resources, in my Definitions, it does not launch an instance for the job. It believes the environment variables are missing, but they get created only on the launch instance that is meant to run the job. Is there some sort of work around for this?
d
Hi Noam - there's a bit of a gotcha here, the secrets also need to be available on your code server task
n
Hey Daniel, is there a best practice of going about this?
d
It's easy to do in ECS in general (you edit the task definition), but docker-compose on ECS has somewhat shockingly bad support for doing it, it seems the only easy way to do it sets them as a file rather than as an environment variable: https://docs.docker.com/cloud/ecs-integration/#secrets
one possibility is to edit the cloudformation template that it creates instead: https://docs.docker.com/cloud/ecs-integration/#tuning-the-cloudformation-template (as per the other post in this channel today, it seems that the docker-compose integration with ECS may not be supported for very much longer anyway)
n
I see, ya this is definitely not ideal. What were they thinking 😄 Thanks for your help Daniel!
I ended up setting empty environment variables for the code server on the docker compose file. That way it still executes with no errors and the instance that runs the job gets the secrets using the dagster.yaml file.