Hi, I found this in the docs but couldn’t find ho...
# ask-community
m
Hi, I found this in the docs but couldn’t find how to actually achieve that (https://docs.dagster.io/concepts/code-locations) - do you have any pointers?
Copy code
Each code location can be sourced from a separate Python environment, so teams can manage their dependencies (or even their Python versions) separately.
I’m looking for ways to have different projects use different virtual environments (and dependencies) while using the same dagster instance - is this possible?
@daniel
d
Here’s a pointer to a tutorial - there are likely others, this is just the first one I found https://realpython.com/python-virtual-environments-a-primer/
m
So my question is specifically on having different dagster code location run on the back of a different environment
I’d like to have separate projects (each of which maps to a git repo, but that’s irrelevant) and load each of them as a code location within the same dagster instance; each project here has its own virtual env. Dagster docs hints that this can be achieved but it’s unclear to me how
d
Er in case that link didn’t work, the example at the bottom here https://docs.dagster.io/concepts/code-locations/workspace-files
m
I saw that - but I don’t see where you can specify the virtual env to run each location on?
d
Each virtual environment has its own python executable path, so the executable_path key determines the virtual environment
We’re talking about this part right? “To enable this use case, Dagster supports customizing the Python environment for each code location by adding the executable_path key to the YAML for a location. These environments can involve distinct sets of installed dependencies, or even completely different Python versions. For example:”
c
Hi @daniel, figured it would be worth tagging you in this thread since it looks like Marco was having the same problem I'm having. I did indeed (after a lot of searching) find the snippet you linked on my own, however I'm left with two concerns: 1. The section you linked notes that "This is applicable only for code locations defined using
@repository
." Is that accurate? Seems like much of this section probably applies to Definitions as well. 2. I feel like I'm piecing together random bits from various different parts of the docs regarding workspaces - for example: location_name and executable_path are only mentioned on this page (which is the Workspace Files docs page) once I click the tab you linked for "Loading multiple repositories," which I initially did NOT click because my thought was "well this isn't relevant to me since I don't plan to use repositories since the docs have been telling me to use the more recent Definitions concept." My point is I'm having a really hard time figuring out how to do something that Dagster claims it is good at: isolating dependencies across projects/teams. So: a. Do you know of any place that has more verbose docs on workspaces and all the possible fields I can specify? Or any really good examples I can parse through? b. If I may suggest - it feels like this is a blind spot in the docs. Surely I'm not the first person who was using Airflow, has had incredible headaches with dependencies, saw that Dagster should resolve them - and then struggled to find out how to do so in the docs. I might even argue that this topic (how to set up Dagster to handle many different dependency requirements) deserves its own docs page. For example: I assume I also need to define those venvs in my build somehow? Such a thing + an example feels worthy of docs.
d
Hey Conor - I think this is really great feedback, any chance you could post the second part in #dagster-feedback to make sure that the right folks on the team (in particular the people thinking about holes in our docs) see it?
c
Yes of course! Do you have any suggestions for me, or will folks over there be better able to help?
d
I think you're correct that loading multiple repositories (should probably say "loading multiple code locations" is just as applicable for Definitions as it is for repositories)
✔️ 1
when you say "I assume I also need to define those venvs in my build somehow" - what are you referring to by 'my build' exactly? What exactly you need to do to get venvs working will probably depend on how you're running dagster (locally / in docker / etc.)
Here's a quick PR that does the rename - "Loading multiple Python environments" is actually probably the right name https://github.com/dagster-io/dagster/pull/12201
🌈 1
c
Right I plan to do some kind of dockerized deployment, although our infra people may ultimately involve kubernetes I assume. But point being I need to ensure all the right venvs exist on my own outside of dagster's configs right? As in, Dagster doesn't deal with creating or maintaining those.
That's great, thank you for doing that so quickly
d
so if you're doing docker or kubernetes - another option that achieves a similar goal is to make a different Docker image for each code location
but yeah with both options Dagster assumes that you have created those environments or images already and are just pointing to them within Dagster
c
Ok yeah I figured, makes enough sense, thanks!
c
Thanks for doing this update, the examples are helpful. I was just about to ask the same question about how to configure those python envs, while looking at the updated page (had the question "how do I do that?" before I finished reading the page and went off to search) - maybe including an executable_path in one of the examples in the "configuring code locations" section would make it more obvious.
m
Thanks Daniel. I was also able to set multiple code location based on the docs, in the end. One problem I have at the moment though is how to set up env variables prior to the ‘executable path’ relative to a given code location being executed. Or more generally running a ‘custom’ executable, e.g. a script that sets env vars prior to calling Python. If I do that I get a ‘[Errno 8] Exec format error’. Any idea how to solve?
d
Would it be possible to make a new post for that question? That way our support oncall will see it
m
👍🏻