https://dagster.io/ logo
Powered by
#dagster-cloud
Title
# dagster-cloud
m

Mark Fickett

12/16/2022, 7:01 PM
I added an environment variable in Dagster Cloud UI, but when I started a job it couldn't find my env var. Do I need to do extra steps (reload code location, update agent to a particular version, etc)? I don't see the env var set on my k8s pod, but I assume it's picked up / published later in the process than that.
d

daniel

12/16/2022, 7:02 PM
Hey Mark - the most likely gotcha here is needing to be on 1.0.17 or later
ty thankyou 1
on both the agent and the image running the code
m

Mark Fickett

12/16/2022, 7:04 PM
The agent is and image should be on 1.1.6, though I can double check the latter.
d

daniel

12/16/2022, 7:04 PM
I can check our logs if you have the deployment name handy
m

Mark Fickett

12/16/2022, 8:57 PM
It's 
td-dev
if you don't mind, thanks!
d

daniel

12/16/2022, 8:58 PM
I do see the right version on that deployment - then the followup questions would be: a) did you check that the code image was also >= 1.0.17 b) how exactly are you checking the env var in code?
👀 1
m

Mark Fickett

12/19/2022, 3:16 PM
Hi Daniel, hope you had a good weekend. I republished the code image and verified it was using 1.1.6 as well; it still failed. Attached are logs/describe for the failed pod. And, I pushed a redeployed the Helm chart for the agent, only changing it to add the env var that I was missing in 
workspace.envVars
and 
dagsterCloudAgent.env
. After adding that to the agent, my job runs fine. The env var I'm looking for is 
PIPELINE_DATABASE
. When I have it set on the agent's chart, I can see it in the pod like:
Copy code
Environment:
      DAGSTER_HOME:                          /opt/dagster/dagster_home
      PIPELINE_DATABASE:                     form_data16
      ...
Screenshot attached for how it looks in the deployment in Dagster Cloud. I'm loading the value using decouple, so it searches in env vars (and then in a 
.env
file if present).
no-env-run-id-8cc72903-e73f-4c65-ac71-eb80082430e8.txt
d

daniel

12/19/2022, 3:20 PM
where are you referencing the environment variable in your code?
I can familiarize myself with decouple - maybe it somehow manages to try to access the environment variables before we inject them into the environment, but that would be surprising
m

Mark Fickett

12/19/2022, 3:24 PM
It should be loading them at import time.
(Also I see your sick status, hope you feel better & no rush to get to this, it's easy enough to work around by just sending my env vars in the agent chart.)
🙏 1
d

daniel

12/19/2022, 3:24 PM
Have you tried accessing the environment variable directly vs using decouple? Just to rule that out?
m

Mark Fickett

12/19/2022, 3:26 PM
I haven't, just since it's some legwork to publish an image that tests that out. If it seems worthwhile I can check that.
d

daniel

12/19/2022, 3:27 PM
No prob - when I’m back fully active, likely later this week, I can try and see if we can reproduce with decouple
Oh, I think I know what the problem is. I think these env vars don’t work when they are needed to load a “system” component like the managed python loggers - that’s likely something we can fix. They should work for loading env vars within your jobs/ops/assets though
Should be a quick fix - we’ll just need to inject the env vars before we initialize the loggers
m

Mark Fickett

12/19/2022, 3:45 PM
Cool!
In case it's useful, I added an 
os.environ.get("PIPELINE_DATABASE")
just before the code that calls decouple and confirmed that that returns 
None
.
d

daniel

12/21/2022, 4:36 AM
https://github.com/dagster-io/dagster/pull/11239 should squash this - was able to reproduce the issue when importing a python logger that needed the env var
thankyou 1
m

Mark Fickett

01/06/2023, 2:28 PM
Did this go into 1.1.8? I don't see it specifically in the release notes, but I assume it would be in the latest release. (I haven't tried out the scenario again.)
d

daniel

01/06/2023, 2:28 PM
It did, yeah
ty thankyou 1
m

Mark Fickett

01/06/2023, 3:21 PM
Great, confirmed that works for me now!
condagster 1
5 Views
Powered by