Hey! How should I specify multiple python environm...
# ask-community
l
Hey! How should I specify multiple python environments in
workplace.yml
for deploy_docker example? When I try adding
Copy code
load_from:
  # - grpc_server:
  #     host: docker_example_pipelines
  #     port: 4000
  #     location_name: "example_pipelines"
  - python_file: 
      relative_path: repo.py
      executable_path:  /opt/conda/envs/test/bin/python
Run queue gives error
Caught an error for run c59de7e6-0b8e-48e4-a45a-86e33ccc0881 while removing it from the queue. Marking the run as failed and dropping it from the queue: Exception: No docker image specified by the instance config or repository
d
Hi Levan - in that example I think you sort of get multiple python environments automatically since each grpc server can use its own image (so you could use a different base image to make Python be a different version in a certain image, or install different sets of requirements in different images)
(so I wouldn't expect you to need to specify an executable_path or switch to
python_file
when using that example - instead, you would most likely change the Dockerfile you use to build the pipeline image
l
Thanks Daniel! Does that mean I have to create separate dockerfiles for each environment I need?
and can I specify which dockerfile should be responsible for specific repository?
d
That’s right, one Dockerfile per environment
And your workspace.yaml uses port numbers to point at the right server/container
So the servers would need to be exposed at those ports as well in the docker compose file
l
hmm, I have ~15 github repos with scripts, that need to be executed in their own environment(with different requirements.txt). So creating 1 for each probably would be too much. Maybe I can dynamically create environments and activate them during pipeline run as the first step?
d
Hi levan - I realized that another option to accomplish this would be to override the command for the different Dockerfiles - i.e. instead of setting the CMD in the Dockerfile like this: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/examples/deploy_docker/Dockerfile_user_code?L23:6 You could set it in the docker-compose file, and vary it per deployment, something like: command: "YOUR_PYTHON_EXECUTABLE -m dagster.api.grpc -h 0.0.0.0 -p 4000 -f repo.py"
so this might actually be possible today without us needing to make any code changes
l
Thanks daniel, will test that as well!
d
Hmmm but then when you go to launch the run it wouldn’t be using that same entry point
l
hi @daniel, getting back to this after some time. So which approach is preferred with latest changes?
d
We have a
--use-python-environment-entry-point
flag now for the
dagster api grpc
command that I think will make it work the way that you want (when runs are launched, they will use the same python environment in which the gRPC server was launched: https://docs.dagster.io/_apidocs/cli#cmdoption-dagster-api-grpc-use-python-environment-entry-point)
(instead of the default
dagster
entry point)
l
Awesome, that works!
a
Hi @daniel, I see that this solution
--use-python-environment-entry-point
is removed since Dagster version 1.0.0. Is it now the case that it gets the python environment of the corresponding gRPC by default? Or has there become a different solution for having multiple python environments? Thank you in advance!
d
Hi Arnoud - that solution should still work, I don't believe it's been removed
a
Oke, it is removed from the documentation at least! But good to know, the solution still works :)
d
Ah you're right - I'll see why that was removed and if we can bring it back
❤️ 1
179 Views