https://dagster.io/ logo
l

Levan

10/19/2021, 10:19 AM
Hey! How should I specify multiple python environments in
workplace.yml
for deploy_docker example? When I try adding
Copy code
load_from:
  # - grpc_server:
  #     host: docker_example_pipelines
  #     port: 4000
  #     location_name: "example_pipelines"
  - python_file: 
      relative_path: repo.py
      executable_path:  /opt/conda/envs/test/bin/python
Run queue gives error
Caught an error for run c59de7e6-0b8e-48e4-a45a-86e33ccc0881 while removing it from the queue. Marking the run as failed and dropping it from the queue: Exception: No docker image specified by the instance config or repository
d

daniel

10/19/2021, 12:52 PM
Hi Levan - in that example I think you sort of get multiple python environments automatically since each grpc server can use its own image (so you could use a different base image to make Python be a different version in a certain image, or install different sets of requirements in different images)
(so I wouldn't expect you to need to specify an executable_path or switch to
python_file
when using that example - instead, you would most likely change the Dockerfile you use to build the pipeline image
l

Levan

10/19/2021, 1:20 PM
Thanks Daniel! Does that mean I have to create separate dockerfiles for each environment I need?
and can I specify which dockerfile should be responsible for specific repository?
d

daniel

10/19/2021, 1:44 PM
That’s right, one Dockerfile per environment
And your workspace.yaml uses port numbers to point at the right server/container
So the servers would need to be exposed at those ports as well in the docker compose file
l

Levan

10/19/2021, 1:54 PM
hmm, I have ~15 github repos with scripts, that need to be executed in their own environment(with different requirements.txt). So creating 1 for each probably would be too much. Maybe I can dynamically create environments and activate them during pipeline run as the first step?
d

daniel

10/26/2021, 2:12 PM
Hi levan - I realized that another option to accomplish this would be to override the command for the different Dockerfiles - i.e. instead of setting the CMD in the Dockerfile like this: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/examples/deploy_docker/Dockerfile_user_code?L23:6 You could set it in the docker-compose file, and vary it per deployment, something like: command: "YOUR_PYTHON_EXECUTABLE -m dagster.api.grpc -h 0.0.0.0 -p 4000 -f repo.py"
so this might actually be possible today without us needing to make any code changes
l

Levan

10/27/2021, 12:54 PM
Thanks daniel, will test that as well!
d

daniel

10/27/2021, 1:20 PM
Hmmm but then when you go to launch the run it wouldn’t be using that same entry point
l

Levan

01/21/2022, 3:56 PM
hi @daniel, getting back to this after some time. So which approach is preferred with latest changes?
d

daniel

01/21/2022, 4:05 PM
We have a
--use-python-environment-entry-point
flag now for the
dagster api grpc
command that I think will make it work the way that you want (when runs are launched, they will use the same python environment in which the gRPC server was launched: https://docs.dagster.io/_apidocs/cli#cmdoption-dagster-api-grpc-use-python-environment-entry-point)
(instead of the default
dagster
entry point)
l

Levan

01/21/2022, 7:26 PM
Awesome, that works!
a

Arnoud van Dommelen

09/20/2022, 8:02 AM
Hi @daniel, I see that this solution
--use-python-environment-entry-point
is removed since Dagster version 1.0.0. Is it now the case that it gets the python environment of the corresponding gRPC by default? Or has there become a different solution for having multiple python environments? Thank you in advance!
d

daniel

09/20/2022, 11:27 AM
Hi Arnoud - that solution should still work, I don't believe it's been removed
a

Arnoud van Dommelen

09/20/2022, 11:53 AM
Oke, it is removed from the documentation at least! But good to know, the solution still works :)
d

daniel

09/20/2022, 12:11 PM
Ah you're right - I'll see why that was removed and if we can bring it back
❤️ 1
19 Views