Hello, I am using multiple code locations in dagst...
# dagster-plus
q
Hello, I am using multiple code locations in dagster cloud and these code locations use different virtual environments. Going through the docs, I see this is possible but I can't seem to figure out where to create the virtual environment. so my
dagster_cloud.yaml
will have something like this
Copy code
load_from:
  - python_file:
      relative_path: path/to/dataengineering_spark_team.py
      location_name: dataengineering_spark_team_py_38_virtual_env
      executable_path: venvs/path/to/dataengineering_spark_team/bin/python
  - python_file:
      relative_path: path/to/team_code_location.py
      location_name: ml_team_py_36_virtual_env
      executable_path: venvs/path/to/ml_tensorflow/bin/python
But where should I put these venv directories be? Should my dockerfile create these and install the packages as part of the build process? We are using a hybrid deployment
j
The standard practice is to create a separate Dockerfile for each code location, so you don’t need multiple virtualenvs in one Dockerfile
In some cases you may want to optimize by only having to build and push 1 dockerfile, etc. For those cases yes you’d want to create the virtualenvs at different paths inside the Dockerfile
q
Interesting! Thanks @johann. I will test this and let you know how that goes