https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Stanley Yang

06/16/2022, 8:17 PM
with a local workspace.yaml, is it possible to use docker repos, similar to user-deployments? From reading the workspaces docs, it seems like my only options are
python_file
and
python_module
. What I am looking for is to replicate a user deployments where I can (container) registry and use the same container for multiple images through different tags. Example config:
Copy code
deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
    - name: "k8s-my-older-user-code-1"
      image:
        repository: "<http://docker.io/dagster/user-code-example|docker.io/dagster/user-code-example>"
        tag: olderSHA
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
I feel like if I want to achieve this, I would need to stand up something like
kind
or
minikube
to replicate my production dagster infrastructure
d

daniel

06/16/2022, 8:19 PM
Hi stanley - we have a "Deploying in Docker" example that works a bit like this: https://docs.dagster.io/deployment/guides/docker#example It requires a little bit more configuration in a couple of places since docker doesn't have a helm to spin up docker containers for you though
s

Stanley Yang

06/16/2022, 8:34 PM
hi @daniel, thanks for the help. I had a look at those docs but it seems like that approach is still limited by
workspace.yml
. Maybe there is some route through launching jobs in containers Is there a full API documentation for the workspace YAML spec? Maybe I can finesse a more creative solution
although at some point, I may just give into deploying k8s locally and replicating the full production stack for localdev
d

daniel

06/16/2022, 8:35 PM
I think it depends on what exactly you mean by 'limited by workspace.yaml'. That example lets you both load your code and launch your jobs in Docker containers
the workspace.yaml file points at a grpc server running in a docker container: https://github.com/dagster-io/dagster/blob/0.15.0/examples/deploy_docker/workspace.yaml#L2-L6
s

Stanley Yang

06/16/2022, 8:40 PM
I want to launch multiple user-code repos. Suppose I have 3 repos: 1.
first_repo_foo.py
in
image:foo
2.
second_repo_bar.py
in
second-image:bar
3.
second_repo_bar.py
in
second-image:bar_other_tag
I want to run all these locally. Can I specify the grpc_server like this?
Copy code
load_from:
  # Each entry here corresponds to a container that exposes a gRPC server.
  - grpc_server:
      host: image:foo
      port: 4000
      location_name: "first_repo_foo.py"
  - grpc_server:
      host: second-image:bar
      port: 4000
      location_name: "second_repo_bar.py"
  - grpc_server:
      host: second-image:bar_other_tag
      port: 4000
      location_name: "second_repo_bar.py"
d

daniel

06/16/2022, 8:41 PM
You can, exactly like that
Each of those hosts will also need to have an entry in the docker-compose file as well
s

Stanley Yang

06/16/2022, 8:43 PM
ah, I'll give it a shot. Thanks!
condagster 1
51 Views