Hey 👋 Can we pass an environment variable to an Op input? It doesn’t seem to be the case, but want to make sure I’m not missing something obvious. For config, we would pass an environment variable as such:

"resources": {
        "snowflake": {
            "config": {
                "account": {"env": "SNOWFLAKE_ACCOUNT"},
                ...
            }
        },

However, the same thing does not seem possible for an input:

"list_objects": {
            "inputs": {
                "bucket": {"env": "S3_BUCKET"},

Hi Charles, I did something similar by defining resources and passing through

@job(resource_defs={...})

to make them available to the Ops, does that sound right?

cc @owen

@Ben Jordan Interesting, that sounds like a potential workaround! We ended up moving

bucket

to a config param for now, but that prevents us from connecting that input to an upstream op if we needed to pass that programmatically in the future 🤔

I think the classic:

"list_objects": {
            "inputs": {
                "bucket": os.getenv("S3_BUCKET"),

would work in this scenario (sorry for the late reply). For more context, the reason the

env

solution doesn't work is that the snowflake schema is defined with a StringSource as the config type for that field (which accepts either a regular string or the env thing, and does the nice conversion). The input field can have a ton of different types and I don't think we special-case the scenario when we detect a string type as the input type from the op (I believe we default to an Any kind of type).

Perfect, thanks @owen!

Hi @owen, so I tried your suggestion above here as input to a mlflow resource that expects a String. I noticed two things: 1. If I write it directly in the launchpad, it treats the command as a string itself 2. If I write it in the

.py

file in which my configs lives, it seems like it cannot get my env variable. It returns None. I specify the environment variable in a

staging.yml

that I use for dagster-cloud deployment.

container_context:
  ecs:
    secrets:

section.

hi @Patrice O'Carroll! in order for this to work, the environment variable would need to be set in the environment that the op is being executed in (not the one that the run is launched from).

Not sure I understand. I tried to deploy the environment (staging) as well as the job itself in staging. But it seems like it checks if the environment variable exists for the configuration before the environment variable is deployed using the .yml file.

oh gotcha, sorry. it is true that the config will be validated in the launching environment and so it does need to be valid there (even if the actual value used only needs to be available in the environment where the op is executed)

ah, ok, thanks for the suggestion!

@Patrice O'Carroll correct me if I’m wrong, but I believe this is to bypass the current limitation in

dagster-mlflow

with passing environment variables to the mlflow resource. This PR I submitted last week would solve the need for relying on

os.getenv

for now if we can get it merged to the next Dagster release 😄 https://github.com/dagster-io/dagster/pull/7997

Yes, I think (hope) this will solve the issue!