Hey :wave: Can we pass an environment variable to ...
# ask-community
c
Hey 👋 Can we pass an environment variable to an Op input? It doesn’t seem to be the case, but want to make sure I’m not missing something obvious. For config, we would pass an environment variable as such:
Copy code
"resources": {
        "snowflake": {
            "config": {
                "account": {"env": "SNOWFLAKE_ACCOUNT"},
                ...
            }
        },
However, the same thing does not seem possible for an input:
Copy code
"list_objects": {
            "inputs": {
                "bucket": {"env": "S3_BUCKET"},
👀 1
b
Hi Charles, I did something similar by defining resources and passing through
@job(resource_defs={...})
to make them available to the Ops, does that sound right?
j
cc @owen
c
@Ben Jordan Interesting, that sounds like a potential workaround! We ended up moving
bucket
to a config param for now, but that prevents us from connecting that input to an upstream op if we needed to pass that programmatically in the future 🤔
o
I think the classic:
Copy code
"list_objects": {
            "inputs": {
                "bucket": os.getenv("S3_BUCKET"),
would work in this scenario (sorry for the late reply). For more context, the reason the
env
solution doesn't work is that the snowflake schema is defined with a StringSource as the config type for that field (which accepts either a regular string or the env thing, and does the nice conversion). The input field can have a ton of different types and I don't think we special-case the scenario when we detect a string type as the input type from the op (I believe we default to an Any kind of type).
c
Perfect, thanks @owen!
🙌 1
p
Hi @owen, so I tried your suggestion above here as input to a mlflow resource that expects a String. I noticed two things: 1. If I write it directly in the launchpad, it treats the command as a string itself 2. If I write it in the
.py
file in which my configs lives, it seems like it cannot get my env variable. It returns None. I specify the environment variable in a
staging.yml
that I use for dagster-cloud deployment.
Copy code
container_context:
  ecs:
    secrets:
section.
o
hi @Patrice O'Carroll! in order for this to work, the environment variable would need to be set in the environment that the op is being executed in (not the one that the run is launched from).
p
Not sure I understand. I tried to deploy the environment (staging) as well as the job itself in staging. But it seems like it checks if the environment variable exists for the configuration before the environment variable is deployed using the .yml file.
o
oh gotcha, sorry. it is true that the config will be validated in the launching environment and so it does need to be valid there (even if the actual value used only needs to be available in the environment where the op is executed)
so i believe if you did os.getenv(“ENVVAR”, default=“”), then this would solve the issue
p
ah, ok, thanks for the suggestion!
Hi @owen, when I tried the suggestion above:
os.getenv("ENVVAR", default="")
it allows me to deploy, sure, but then in the launchpad it still does not fetch my env variable, it uses the default empty string instead. I define my environment variable using the (new?) feature to add them in the code location following this syntax;
Copy code
container_context:
  ecs:
    secrets:
Could that be the issue? See the
ml-models
code location file here https://dribbble.dagster.cloud/staging/workspace
c
@Patrice O'Carroll correct me if I’m wrong, but I believe this is to bypass the current limitation in
dagster-mlflow
with passing environment variables to the mlflow resource. This PR I submitted last week would solve the need for relying on
os.getenv
for now if we can get it merged to the next Dagster release 😄 https://github.com/dagster-io/dagster/pull/7997
p
Yes, I think (hope) this will solve the issue!