Hi support After searching the docs and on slack I am still

Question

Hi support After searching the docs and on slack I am still quite confused on how to implement multiple python environments for dagster on k8s I am guessing there is need to update each workspace docker container with the required python environments update the user deployment yaml file with `use python environment entry point` argument for the grpc somewhere somehow give dagster which environment relates to which dagster repository Would anyone have some example setup to share

daniel · Answer

Hi Greg the simplest way to run N different Python environments in k8s is to build N different Docker images each with a different set of python dependencies installed and put each one in it own user deployment <https docs dagster io deployment guides kubernetes deploying with helm configure your user deployment> Is that an option for you

Greg · Answer

Hi < daniel> Thanks for answering It does not seem like an option to me At least based on how I see the architecture of our projects If I would create a user development for each dagster repository I would end up with 100s of them for now and surely 1000s in the future It does not seem very manageable to me Instead we are creating a user development for each of our client containing all the related dagster repositories But of course each repository would ideally use different python environment Does it make sense

daniel · Answer

It does make sense right now dagster requires that each user code deployment have exactly one python environment though

daniel · Answer

Since it loads that code in a Python process and there isn t a way for a python process to have multiple Python environments

Greg · Answer

I am quite confused to what is multiple python environment then < daniel>

daniel · Answer

In that example each of those still has its own process there s a server for each one

daniel · Answer

each one of the entries in that workspace yaml corresponds to a pod if you re using the Dagster helm chart

Greg · Answer

Ok I see

Greg · Answer

So there is no way around I guess then That s a bit of a pain point for us But thank you for answering < daniel>

daniel · Answer

No problem this has come up a bit before hundreds of different python environments resulting in a lot of servers and its not a use case we have a fantastic answer for currently Do you only need the jobs to be in different Python environments when the steps execute Dagster loads your code both to load the jobs and display them in dagit and then also again when running the actual job

Greg · Answer

Yes only for job execution < daniel>

Greg · Answer

Would it for instance work to update the shebang to point to a virtual environment created on the container < daniel>

Greg · Answer

Or being able to configure the `execution path` for each job through could do the trick since it is a argument sent to the job config in k8s < daniel> ``` entry point dagster executable path usr local bin python pipeline run id fc7dafdc 95b6 4486 a41d c7328a575c6a set exit code on failure null ```

Greg · Answer

But you were saying that the problem would come from loading