I have a solid that sends files to multiple locations. How can I configure these paths using environment variables?

hi @Martim Passos! The built-in Dagster config type that provides the ability to configure via env variables is StringSource. You could write your own custom config type to give you the behavior you show in the screenshot, or if you want to use the built-ins, it would look something like:

# config schema
@solid(config_schema={'backlog': StringSource, 'jpeg_hd': StringSource}):
def ...

# config yaml
file_dispatcher:
  backlog:
    config:
       env: IMG_BACKLOG
  jpeg_hd:
    config:
       env: JPEG_HD

Thanks @owen, I got it to work with minor modifications to your example: Dagster requires the key “config” after the solid’s name, so the working result looks like this:

ah I see what you mean -- super reasonable assumption. glad you got it working !

lol the image I sent was wrong. Here’s the correct one:

haha I figured

Hi @owen I'm just trying to switch over to use StringSource from using hard coded env variables in yaml.

9 @solid(config_schema={                                                          
 160             'cronjob_user': StringSource,                                       
 161             'cronjob_password': StringSource                                    
 162         })                                                                      
 163 def authenticate_api(context: SolidExecutionContext) -> dict:                   
 164     """                                                                         
 165     Given cronjob_user and cronjob_password returns json_headers containing     
 166     authorization token                                                         
 167     """                                                                         
 168     cronjob_user = context.solid_config['cronjob_user']                         
 169     cronjob_password = context.solid_config['cronjob_password']

When I print what Dagster things cronjob user is within the pipeline I just get CRONJOB_USER (the input in the yaml as opposed to the corresponding env variable) where am I going wrong?

hm I'm not as familiar with the DockerRunLauncher stuff (cc @johann). just for more context, I think the original issue you were seeing would be caused by passing in

{"config":"CRONJOB_USER"}

instead of

{"config":{"env":"CRONJOB_USER"}}

because StringSource accepts either a normal string or the name of an env var.

@George Pearse mind starting a new thread for the docker env question

@johann apologies for the spam, some deadlines are approaching, Alex sorted me out here https://dagster.slack.com/archives/C01U954MEER/p1631123244318800

No worries, glad it was resolved