I added an environment variable in Dagster Cloud U...
# dagster-plus
I added an environment variable in Dagster Cloud UI, but when I started a job it couldn't find my env var. Do I need to do extra steps (reload code location, update agent to a particular version, etc)? I don't see the env var set on my k8s pod, but I assume it's picked up / published later in the process than that.
Hey Mark - the most likely gotcha here is needing to be on 1.0.17 or later
ty thankyou 1
on both the agent and the image running the code
The agent is and image should be on 1.1.6, though I can double check the latter.
I can check our logs if you have the deployment name handy
if you don't mind, thanks!
I do see the right version on that deployment - then the followup questions would be: a) did you check that the code image was also >= 1.0.17 b) how exactly are you checking the env var in code?
👀 1
Hi Daniel, hope you had a good weekend. I republished the code image and verified it was using 1.1.6 as well; it still failed. Attached are logs/describe for the failed pod. And, I pushed a redeployed the Helm chart for the agent, only changing it to add the env var that I was missing in
. After adding that to the agent, my job runs fine. The env var I'm looking for is
. When I have it set on the agent's chart, I can see it in the pod like:
Copy code
      DAGSTER_HOME:                          /opt/dagster/dagster_home
      PIPELINE_DATABASE:                     form_data16
Screenshot attached for how it looks in the deployment in Dagster Cloud. I'm loading the value using decouple, so it searches in env vars (and then in a
file if present).
where are you referencing the environment variable in your code?
I can familiarize myself with decouple - maybe it somehow manages to try to access the environment variables before we inject them into the environment, but that would be surprising
It should be loading them at import time.
(Also I see your sick status, hope you feel better & no rush to get to this, it's easy enough to work around by just sending my env vars in the agent chart.)
🙏 1
Have you tried accessing the environment variable directly vs using decouple? Just to rule that out?
I haven't, just since it's some legwork to publish an image that tests that out. If it seems worthwhile I can check that.
No prob - when I’m back fully active, likely later this week, I can try and see if we can reproduce with decouple
Oh, I think I know what the problem is. I think these env vars don’t work when they are needed to load a “system” component like the managed python loggers - that’s likely something we can fix. They should work for loading env vars within your jobs/ops/assets though
Should be a quick fix - we’ll just need to inject the env vars before we initialize the loggers
In case it's useful, I added an
just before the code that calls decouple and confirmed that that returns
https://github.com/dagster-io/dagster/pull/11239 should squash this - was able to reproduce the issue when importing a python logger that needed the env var
thankyou 1
Did this go into 1.1.8? I don't see it specifically in the release notes, but I assume it would be in the latest release. (I haven't tried out the scenario again.)
It did, yeah
ty thankyou 1
Great, confirmed that works for me now!
condagster 1