Hello again, team! How do you usually provide sensitive resources configuration in case of schedules/sensors? Example: We have dev and prod clusters running Dagster, and we want to use different base urls for a http client resource. That's easily resolved in Dagit as we have separate instances of it for dev and prod. But in case of schedules/sensors the configuration is hardcoded. I could use

os.environ

in schedule definition, but I can't do that within Dagit, which will also lead to some approach discrepancy.

Hi Nick, you can use a StringSource type that tells Dagster to load config using an env var, e.g. here's an example:

@op(
    config_schema={
        "foo": Field(StringSource)
    },
)
def my_op(context):
    return context.solid_config["foo"]

Then you could launch the run with e.g this config.

ops:
  my_op:
    env:
      MY_ENV_VAR_NAME

Then you need to make sure that MY_ENV_VAR_NAME is present in the environment where the run is launched (e.g. by configuring your run launcher to include it)

@daniel thanks a lot for your answer, I've just tested that. Some notes: • Not all the values for Dagster provided resources are typed as StringSource, e.g.

s3_resource

has

profile_name

which is Field(str). I think that can overcome that with

@configured

on my side though... • The original problem is a kind of discrepancy between scheduler configuration and dagit one. I can tweak dagit configuration on the fly, e.g. i can add or remove values between runs. Good example is:

resources:
  s3:
    config:
      endpoint_url:
        env: MINIO_ENDPOINT_URL

Usually we need this parameter only for local development with minio. We can easily add it for local dagit and avoid it for production dagit (where we don't even have this env variable set). But that becomes a real problem when we are trying to configure

@schedule

with the same piece of configuration, as its will be bundled into the code, and the env variable will become a requirement.

Got it - it's true that there isn't really a way to change schedule configuration on the fly currently. I could imagine building something on top of the current system that pulls the config from a local YAML file or something rather than from code, and swapping in a new version of that file or something - we don't have anything like that built-in currently though