Hey! How should I specify multiple python environm...
# ask-community
Hey! How should I specify multiple python environments in
for deploy_docker example? When I try adding
Copy code
  # - grpc_server:
  #     host: docker_example_pipelines
  #     port: 4000
  #     location_name: "example_pipelines"
  - python_file: 
      relative_path: repo.py
      executable_path:  /opt/conda/envs/test/bin/python
Run queue gives error
Caught an error for run c59de7e6-0b8e-48e4-a45a-86e33ccc0881 while removing it from the queue. Marking the run as failed and dropping it from the queue: Exception: No docker image specified by the instance config or repository
Hi Levan - in that example I think you sort of get multiple python environments automatically since each grpc server can use its own image (so you could use a different base image to make Python be a different version in a certain image, or install different sets of requirements in different images)
(so I wouldn't expect you to need to specify an executable_path or switch to
when using that example - instead, you would most likely change the Dockerfile you use to build the pipeline image
Thanks Daniel! Does that mean I have to create separate dockerfiles for each environment I need?
and can I specify which dockerfile should be responsible for specific repository?
That’s right, one Dockerfile per environment
And your workspace.yaml uses port numbers to point at the right server/container
So the servers would need to be exposed at those ports as well in the docker compose file
hmm, I have ~15 github repos with scripts, that need to be executed in their own environment(with different requirements.txt). So creating 1 for each probably would be too much. Maybe I can dynamically create environments and activate them during pipeline run as the first step?
Hi levan - I realized that another option to accomplish this would be to override the command for the different Dockerfiles - i.e. instead of setting the CMD in the Dockerfile like this: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/examples/deploy_docker/Dockerfile_user_code?L23:6 You could set it in the docker-compose file, and vary it per deployment, something like: command: "YOUR_PYTHON_EXECUTABLE -m dagster.api.grpc -h -p 4000 -f repo.py"
so this might actually be possible today without us needing to make any code changes
Thanks daniel, will test that as well!
Hmmm but then when you go to launch the run it wouldn’t be using that same entry point
hi @daniel, getting back to this after some time. So which approach is preferred with latest changes?
We have a
flag now for the
dagster api grpc
command that I think will make it work the way that you want (when runs are launched, they will use the same python environment in which the gRPC server was launched: https://docs.dagster.io/_apidocs/cli#cmdoption-dagster-api-grpc-use-python-environment-entry-point)
(instead of the default
entry point)
Awesome, that works!
Hi @daniel, I see that this solution
is removed since Dagster version 1.0.0. Is it now the case that it gets the python environment of the corresponding gRPC by default? Or has there become a different solution for having multiple python environments? Thank you in advance!
Hi Arnoud - that solution should still work, I don't believe it's been removed
Oke, it is removed from the documentation at least! But good to know, the solution still works :)
Ah you're right - I'll see why that was removed and if we can bring it back
❤️ 1