How do make multiple repo files available from the dagster grpc server? •

dagster api grpc -h 0.0.0.0 -p 4000 -f repo1.py

Do I just have multiple -f switch or what?

I believe you have to stand up a separate grpc server for each repo

You can't pass in multiple files, but you can pass in a module that imports the repos from both of the files

@daniel @prha I'm following https://docs.dagster.io/deployment/guides/docker#multi-container-docker-deployment

Here's an example project that loads code from a module: https://github.com/dagster-io/dagster/tree/master/examples/hacker_news If you'd like to be able to reload them independently, then putting them in separate entries in the workspace.yaml makes sense (that would need to be two containers/grpc servers if you're following the example)

@daniel so I would have something like this?

CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "bb_repository.py"]

that looks right to me, yeah

inside the Dockerfile_user_code (following the example)

If you want, you could use the same image/Dockerfile that contains both files, then include the command in the docker-compose file instead of including CMD in the dockerfile

@daniel I'm still trying to understand how the containers interact with each other. Could you give me a pseudocode on what would the CMD look like on the docker-compose file for multiple grpc server/repos?

sure. Do you want to use a single image for both servers? Or a separate image/Dockerfile for each one?

Let's start with single image. I'm pretty sure the moment I said this, there will be the case for two images LOL!

something like this then (after taking the EXPOSE and CMD out of the Dockerfile)

docker_example_user_code:
    build:
      context: .
      dockerfile: ./Dockerfile_user_code
    container_name: docker_example_user_code
    image: docker_example_user_code_image
    restart: always
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
      DAGSTER_CURRENT_IMAGE: "docker_example_user_code_image"
    command: dagster api grpc -h 0.0.0.0 -p 4000 -f repo1.py
    ports:
      - "4000:4000"
    networks:
      - docker_example_network

  docker_other_example_user_code:
    build:
      context: .
      dockerfile: ./Dockerfile_user_code
    container_name: docker_other_example_user_code
    image: docker_example_user_code_image
    restart: always
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
      DAGSTER_CURRENT_IMAGE: "docker_example_user_code_image"
    command: dagster api grpc -h 0.0.0.0 -p 4000 -f repo2.py
    ports:
      - "4001:4001"
    networks:
      - docker_example_network

I see, so as long as the actual repo1.py and repo2.py is in that one image, you can spawn multiple grpc servers pointing to any of them

yeah

so in theory, the grpc server could be remote?

in theory, yeah (more common in e.g. kubernetes or ECS, where it's probably on some other server in the same cluster)

remote as in living in a separate EC2 but accessible via some port (4003) for instance

yeah, definitely possible

Beautiful!

that's another benefit of running the server separately that i should have mentioned, you can redeploy code without needing to restart or rebuild the whole service

But I still have to manually "Reload" on Dagit to get the latest, right?

It'll actually automatically update for you

How would it know when to refresh?

It checks in with the server periodically

heartbeat, isn't it?

I'll definitely surface that - what exactly is the 'this' that unlocked it for you?

We are a data shop that orchestrate many ETLs for different clients. Of course those ETLs are spread through many servers (EC2s)

ah I see, the ability to run code servers on remote instances. got it

So I've been looking for a way to have one nice UI (like Dagit) which will allow me to see those separate ETLs

would you mind making a new post for this? Not a dbt expert

oh sorry. absolutely

Yeah, the default compute logs write to the filesystem, which dagit doesn't have access to

@daniel I have setup the dagster.yaml as such:

compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir: /opt/dagster/dagster_home/storage

And map a volume to the host's filesystem in the docker-compose.yaml:

volumes: # Make docker client accessible so we can launch containers using host docker
      - /var/run/docker.sock:/var/run/docker.sock
      - ./storage:/opt/dagster/dagster_home/storage

I map the above in both dagit and dagster_daemon containers. But.. when I run a job, the raw log view is still empty, the bottom says:

/opt/dagster/dagster_home/storage/08dfbcc7-b8e5-4cb1-890c-9b83b7df5ebd/compute_logs/some_op.out

but I can't find that .out file in the volume (which is why the view is empty). The whole directory is there (up to /compute_logs), just the file is missing. Any thoughts?

the problem is that those mounted volumes aren't included in the containers that are spun up by the run launcher

@daniel this one seems to work: