Hey! How should I specify multiple python environments in

workplace.yml

for deploy_docker example? When I try adding

load_from:
  # - grpc_server:
  #     host: docker_example_pipelines
  #     port: 4000
  #     location_name: "example_pipelines"
  - python_file: 
      relative_path: repo.py
      executable_path:  /opt/conda/envs/test/bin/python

Run queue gives error

Caught an error for run c59de7e6-0b8e-48e4-a45a-86e33ccc0881 while removing it from the queue. Marking the run as failed and dropping it from the queue: Exception: No docker image specified by the instance config or repository

Hi Levan - in that example I think you sort of get multiple python environments automatically since each grpc server can use its own image (so you could use a different base image to make Python be a different version in a certain image, or install different sets of requirements in different images)

Thanks Daniel! Does that mean I have to create separate dockerfiles for each environment I need?

That’s right, one Dockerfile per environment

hmm, I have ~15 github repos with scripts, that need to be executed in their own environment(with different requirements.txt). So creating 1 for each probably would be too much. Maybe I can dynamically create environments and activate them during pipeline run as the first step?

Hi levan - I realized that another option to accomplish this would be to override the command for the different Dockerfiles - i.e. instead of setting the CMD in the Dockerfile like this: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/examples/deploy_docker/Dockerfile_user_code?L23:6 You could set it in the docker-compose file, and vary it per deployment, something like: command: "YOUR_PYTHON_EXECUTABLE -m dagster.api.grpc -h 0.0.0.0 -p 4000 -f repo.py"

Thanks daniel, will test that as well!

Hmmm but then when you go to launch the run it wouldn’t be using that same entry point

hi @daniel, getting back to this after some time. So which approach is preferred with latest changes?

We have a

--use-python-environment-entry-point

flag now for the

dagster api grpc

command that I think will make it work the way that you want (when runs are launched, they will use the same python environment in which the gRPC server was launched: https://docs.dagster.io/_apidocs/cli#cmdoption-dagster-api-grpc-use-python-environment-entry-point)

Awesome, that works!

Hi @daniel, I see that this solution

--use-python-environment-entry-point

is removed since Dagster version 1.0.0. Is it now the case that it gets the python environment of the corresponding gRPC by default? Or has there become a different solution for having multiple python environments? Thank you in advance!

Hi Arnoud - that solution should still work, I don't believe it's been removed

Oke, it is removed from the documentation at least! But good to know, the solution still works :)

Ah you're right - I'll see why that was removed and if we can bring it back