Hey all. We are trying to deploy a pipeline to aws using s3 for intermediate storage. We are following the instructions here (https://docs.dagster.io/0.7.9/docs/deploying/aws, we are using 0.7.9 for the time being) but we need to use an env variable for the bucket name (different bucket names for dev, qa, prod, etc.). I see there is support for env variables in other config areas but it does not seem supported for s3 intermediate storage. We are getting the error

dagster.core.errors.DagsterInvalidConfigError: Error in config for pipeline recommender_pipeline
    Error 1: Invalid scalar at path root:storage:s3:config:s3_bucket. Value "{'env': 'STORAGE_BUCKET'}" of type "<class 'dict'>" is not valid for expected type "String".
make: *** [run] Error 1

with our yaml:

storage:
  s3:
    config:
      s3_bucket:
        env: STORAGE_BUCKET
      s3_prefix: temp
loggers:
  console:
    config:
      log_level: DEBUG

I can exec into the container and see that the

STORAGE_BUCKET

env var is set correctly. If the answer is we just need to update to the latest version of Dagster, then that's what we can do but ideally we would like to know if there is an easy fix for us in the short-term. Hopefully, I am just missing something very simple due to Friday fried-brain 🙂.

this was changed from

str

to

StringSource

(which is what supports the

env

load) in

00a5957a4c4f

which was release first in

0.8.0

Perfect! Thank you!

Also; you should upgrade to the sweet awesomeness that is 0.10 :)