https://dagster.io/ logo
#dagster-plus
Title
# dagster-plus
b

Ben Latz

08/24/2022, 2:08 PM
Really excited to start using the hosted version! Just stuck on custom environment variables import. Using Hybrid w/ ECS and haven’t figured out where to import custom env variables (i’ve tried the AWS Secrets approach with dagster tag, as well as inserting via Github Action secrets in the build file, as well as manually revising the task definition to add env variables but those get overwritten with a fresh task build). Any docs I’ve found re: ECS seem to refer to the Open Source version, which I figure is slightly different (as i deployed via the hybrid cloudformation stack)
j

Joe

08/24/2022, 7:31 PM
hey @Ben Latz AWS secretsmanager secrets can be configured in Hybrid w/ ECS using the 'container_context' field described here: https://docs.dagster.cloud/guides/adding-code#environment-specific-config For example, you could configure your code as follows:
Copy code
container_context:
  ecs:
    secrets:
      - name: "MY_API_TOKEN"
        valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:token::"
      - name: "MY_PASSWORD"
        valueFrom: "arn:aws:secretsmanager:us-east-1:123456789012:secret:FOO-AbCdEf:password::"
(or to add all secrets with a given tag in secretsmanager):
Copy code
container_context:
  ecs:
    secrets_tags:
      - my_secrets_tag
b

Ben Latz

08/24/2022, 7:32 PM
ahh ok got it - saw that but didn’t connect it all the way, thanks! Is this going to GET from secrets only on a deploy or every time the job runs? Secrets gets really expensive for high frequency stuff, so we have typically opted towards using Parameter Store instead
j

Joe

08/24/2022, 7:35 PM
The agent will fetch secrets for every run. We should be able to add support for parameter store though.
b

Ben Latz

08/24/2022, 7:37 PM
cool, yeah the ideal target in my mind is being able to make the env variables part of the container definition so that way it’s just a normal environment variable and doesn’t make get requests every time to AWS’ services
no problem using this in the meantime though - thx for the reply
j

Joe

08/24/2022, 7:39 PM
thanks for the feedback! I don't think theres any major reason why we don't support SSM just haven't got around to it yet.
👍 1
z

Zach

08/25/2022, 6:15 PM
@Ben Latz it's not quite environment variables but I use SSM parameters quite heavily for configuration and wrote a little SSMSource config field that allows you to reference a SSM param in your run config, which gets looked up and automatically replaced with the SSM parameter value when you launch the run - https://gist.github.com/zyd14/1280e424e32d81e5bfb74eb81913706a
7 Views