Anton Friberg

02/08/2023, 1:52 PM
Hi! How would I set up a Dagster resource so that I can have a configured instance of it where configuration is taken from environment variables if they are provided otherwise it takes default or simply None value?
from dagster import resource, Field, StringSource

    "user": Field(StringSource, description="Username."),
    "pass": Field(StringSource, description="Password."),
    "proxy": Field(StringSource, is_required=False, description="HTTP proxy to use if provided."),
def service_credentials(context):
    """S3FileSystem Dagster Resource."""
    return context.resource_config

service_credentials_from_env = service_credentials.configured({
    "user": {"env": "SERVICE_USER"},
    "pass": {"env": "SERVICE_PASS"},
    "proxy": {"env": "SERVICE_HTTP_PROXY"},
If I do the case above it simply errors with
dagster._core.errors.DagsterInvalidConfigError: Error in config for resource service_credentials
    Error 1: Post processing at path root:config:proxy of original value {'env': 'SERVICE_HTTP_PROXY'} failed:
dagster._config.errors.PostProcessingError: You have attempted to fetch the environment variable "SERVICE_HTTP_PROXY" which is not set. In order for this execution to succeed it must be set in this environment.
:dagster-bot-resolve: 1
Ah, it seems even default value fallback is not supported. This really takes away the value from StringSource, IntSource and BoolSource IMO.


02/08/2023, 2:50 PM
Hi Anton, I agree this is a problem-- I assigned this issue to myself and will try to knock it out in the next couple weeks.
❤️ 3

Anton Friberg

02/08/2023, 3:06 PM
Thanks Sean!