Hello there! Got a question about the gRPC server command. Recently changed our project structure such that our

dagster api grpc

command is as follows:

dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-d", "context_folder", "-f", "context_folder/job.py

on trying to launch a run, we are greeted w/ the following error:

dagster.core.errors.DagsterLaunchFailedError: Error during RPC setup for executing run: dagster.core.errors.DagsterRunNotFoundError: gRPC server could not load run 70b65f83-11d7-419c-ac20-5729df3b802b in order to execute it. Make sure that the gRPC server has access to your run storage.

Previously, we had not set a

-d

parameter. the structure of that context_folder is:

context_folder/
-> job.py
-> ops.py

So I suppose my question is: does the dagster.yaml file have to be present in the working directory? I am looking at the docs for dagster api grpc, and i do not see any parameter to supply the location of the dagster.yaml..

Hi Jose - dagster.yaml doesn't need to be present anywhere in the grpc server (we actually pass in a serialized representation of your dagster.yaml when launching the run). However, the grpc server process or container does need to be able to access the storage that's specified in your dagster.yaml. A common way of enabling this when you're running in a separate container is to use postgres for your storage

very interesting..i will continue to investigate and see whether any of my config files have been changed in some significant way.

hello, I am getting the same error after upgrading dagster from 0.13.13 to 0.14.12. the error comes when i try to launch a run, same as Jose. are there any changes I need to make as part of this upgrade? I couldn't find anything new in the ucd Dockerfile/code example in the docs

Hi Josh - I'm not aware of a change between those versions that would cause this issue - as far as I know, 0.13.13 also needed to be able to reach your storage from your user code deployment if you're using the default run launcher. Would you mind sharing your dagster.yaml, and are you expecting that your user code container has access to your run storage? Did any other changes happen other than upgrading versions?

only change is the version change of the dagster services and of the dagster library used in the ucd.

$ cat ucds/dagster.yaml
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_POSTGRES_USERNAME
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      hostname:
        env: DAGSTER_POSTGRES_HOST
      db_name:
        env: DAGSTER_POSTGRES_DATABASE
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_POSTGRES_USERNAME
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      hostname:
        env: DAGSTER_POSTGRES_HOST
      db_name:
        env: DAGSTER_POSTGRES_DATABASE
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_POSTGRES_USERNAME
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      hostname:
        env: DAGSTER_POSTGRES_HOST
      db_name:
        env: DAGSTER_POSTGRES_DATABASE
      port: 5432

# <https://docs.dagster.io/concepts/logging/python-logging>
python_logs:
  python_log_level: INFO
  managed_python_loggers:
    - root
  dagster_handler_config:
    handlers:
      myHandler:
        class: logging.StreamHandler
        level: INFO
        stream: <ext://sys.stdout>
        formatter: detail
    formatters:
      detail:
        format: "%(asctime)s - %(levelname)s - %(name)s - %(module)s - %(message)s"

And this is based on the docker example?

this is my own code. ah i think i sees it! those var names are different

We don't assume those env vars anywhere as far as i know - whatever is put in the dagster.yaml is what it uses

damn

the big mystery i think is why it was working before

i possible, i will double check. lots of config floating around

when you say "writes stuff to" - what kind of stuff?

last one here as i think this still qualifies as relevant [sorry OP] i found the source of my error - the dagster image we build for running dagit etc was missing the dagster.yaml for 0.14.12 headbang easy fix. thanks for your help Daniel. re the permissions issue, looks like at least logs:

PermissionError: [Errno 13] Permission denied: '/opt/dagster'
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/api.py", line 775, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/usr/local/lib/python3.7/site-packages/dagster/core/executor/in_process.py", line 50, in execute
    output_capture=plan_context.output_capture,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/api.py", line 871, in __iter__
    pipeline_context=self.pipeline_context,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/execute_plan.py", line 63, in inner_plan_execution_iterator
    step_context.pipeline_run, step_context.step.key
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/compute_log_manager.py", line 55, in watch
    with self._watch_logs(pipeline_run, step_key):
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/local_compute_log_manager.py", line 49, in _watch_logs
    with mirror_stream_to_file(sys.stdout, outpath):
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/compute_logs.py", line 28, in mirror_stream_to_file
    ensure_file(filepath)
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/__init__.py", line 319, in ensure_file
    ensure_dir(os.path.dirname(path))
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/__init__.py", line 312, in ensure_dir
    os.makedirs(file_path)
  File "/usr/local/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/local/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)

ah got it. that looks like compute logs - you can see how to configure that here: https://docs.dagster.io/deployment/dagster-instance#compute-log-storage

perfect, thanks again!