Hello Does anyone have experience in using

Question

Hello Does anyone have experience in using <https github com theskumar python dotenv|dotenv> while running a local agent I m not sure how to integrate the `load env` call before dagster is booting

daniel · Answer

Hi Raphaël I m not deeply familiar with dotenv but looking through the docs I think load dotenv call inline right in the Python file that defines your Dagster code before any code that needs those environment variables to be available runs could do the trick

Raphaël Riel · Answer

I ll try So somewhere in the Repository file should make it

daniel · Answer

Yeah I think early on in the file or module that you re telling dagster to use to load your code

Raphaël Riel · Answer

Thanks Daniel

daniel · Answer

np let me know how it goes I haven t tried this myself so it s possible there will be some gotchas

Raphaël Riel · Answer

Seems like everything is working except for `DAGSTER HOME` when put in the repository s file

Raphaël Riel · Answer

This causes Dagster to use a new storage location for every invocation

Raphaël Riel · Answer

Which might not be a problem for local dev

daniel · Answer

Hm strange when you say it isn t working you re putting DAGSTER HOMe in your dotenv file but its getting ignored or overwritten

Raphaël Riel · Answer

Ignored

Raphaël Riel · Answer

Probably that having dot env operate from the repo it s added to Env after Dagster is trying to read it

Raphaël Riel · Answer

I also tried putting it in `os environ` from the repo code bypassing dot env Same result

daniel · Answer

Hmmm `dagster job execute` actually isn t technically supported in dagster cloud currently although it appears it is working in some fashion Does the same problem happen if you launch a run from within Dagit in Dagster Cloud

Raphaël Riel · Answer

Let me see

Raphaël Riel · Answer

Looks like the Env is picked up by Dagster when I run an agent from my machine and execute a Run from the Cloud

daniel · Answer

ah great

daniel · Answer

We should add a replacement for `dagster job execute` for cloud I ll make sure that gets tracked

Raphaël Riel · Answer

What would you suggest I do to test my jobs straight from my CLI

daniel · Answer

and probably have it fail if you try to use it in cloud instead of mostly work

Raphaël Riel · Answer

I like the idea of launching a run straight from my IDE s terminal

daniel · Answer

Yeah it s an entirely reasonable request

Raphaël Riel · Answer

And potentially having a debugger attached

Raphaël Riel · Answer

Is this a valid way of doing such a thing

Raphaël Riel · Answer

It s working

daniel · Answer

I was about to recommend that as a slightly unsatisfying alternative yeah slightly smiling face

Raphaël Riel · Answer

lol

Raphaël Riel · Answer

What s the caveats

daniel · Answer

Although I think setting DAGSTER HOME separately outside of the call to `dagster job execute` call would work as well

daniel · Answer

i e setting in the system environment I think the problem you re running into is that DAGSTER HOME is really a dagster system env var so by the time you load it in your job code it s too late

Raphaël Riel · Answer

Exact That was another option I m willing to try

daniel · Answer

The only caveat is that its annoying to have to write your own script vs using a CLI

Raphaël Riel · Answer

I m actually ok with adding the ` main ` thing in file that contains graph jobs I want to run locally for development purposes

daniel · Answer

Ah OK in that case that sounds great execute in process is 100 supported in cloud

Raphaël Riel · Answer

And when the file is loaded by the repo it should skip this `if` anyway and act as a normal agent

Raphaël Riel · Answer

Since the file won t be executed directly