https://dagster.io/ logo
#ask-community
Title
# ask-community
j

Jose Uribe

05/05/2022, 5:50 PM
Hello there! Got a question about the gRPC server command. Recently changed our project structure such that our
dagster api grpc
command is as follows:
Copy code
dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-d", "context_folder", "-f", "context_folder/job.py
on trying to launch a run, we are greeted w/ the following error:
Copy code
dagster.core.errors.DagsterLaunchFailedError: Error during RPC setup for executing run: dagster.core.errors.DagsterRunNotFoundError: gRPC server could not load run 70b65f83-11d7-419c-ac20-5729df3b802b in order to execute it. Make sure that the gRPC server has access to your run storage.
Previously, we had not set a
-d
parameter. the structure of that context_folder is:
Copy code
context_folder/
-> job.py
-> ops.py
So I suppose my question is: does the dagster.yaml file have to be present in the working directory? I am looking at the docs for dagster api grpc, and i do not see any parameter to supply the location of the dagster.yaml..
d

daniel

05/05/2022, 6:03 PM
Hi Jose - dagster.yaml doesn't need to be present anywhere in the grpc server (we actually pass in a serialized representation of your dagster.yaml when launching the run). However, the grpc server process or container does need to be able to access the storage that's specified in your dagster.yaml. A common way of enabling this when you're running in a separate container is to use postgres for your storage
j

Jose Uribe

05/05/2022, 6:04 PM
very interesting..i will continue to investigate and see whether any of my config files have been changed in some significant way.
Ah ok I've identified the issue; wasn't copying the file in my docker image to the DAGSTER_HOME directory. Thanks for the clarification on grpc servers!
condagster 1
j

Josh Karlin

05/12/2022, 2:07 PM
hello, I am getting the same error after upgrading dagster from 0.13.13 to 0.14.12. the error comes when i try to launch a run, same as Jose. are there any changes I need to make as part of this upgrade? I couldn't find anything new in the ucd Dockerfile/code example in the docs
d

daniel

05/12/2022, 2:13 PM
Hi Josh - I'm not aware of a change between those versions that would cause this issue - as far as I know, 0.13.13 also needed to be able to reach your storage from your user code deployment if you're using the default run launcher. Would you mind sharing your dagster.yaml, and are you expecting that your user code container has access to your run storage? Did any other changes happen other than upgrading versions?
j

Josh Karlin

05/12/2022, 2:37 PM
only change is the version change of the dagster services and of the dagster library used in the ucd.
Copy code
$ cat ucds/dagster.yaml
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_POSTGRES_USERNAME
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      hostname:
        env: DAGSTER_POSTGRES_HOST
      db_name:
        env: DAGSTER_POSTGRES_DATABASE
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_POSTGRES_USERNAME
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      hostname:
        env: DAGSTER_POSTGRES_HOST
      db_name:
        env: DAGSTER_POSTGRES_DATABASE
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username:
        env: DAGSTER_POSTGRES_USERNAME
      password:
        env: DAGSTER_POSTGRES_PASSWORD
      hostname:
        env: DAGSTER_POSTGRES_HOST
      db_name:
        env: DAGSTER_POSTGRES_DATABASE
      port: 5432

# <https://docs.dagster.io/concepts/logging/python-logging>
python_logs:
  python_log_level: INFO
  managed_python_loggers:
    - root
  dagster_handler_config:
    handlers:
      myHandler:
        class: logging.StreamHandler
        level: INFO
        stream: <ext://sys.stdout>
        formatter: detail
    formatters:
      detail:
        format: "%(asctime)s - %(levelname)s - %(name)s - %(module)s - %(message)s"
d

daniel

05/12/2022, 2:38 PM
And this is based on the docker example?
The line in the docker example that ensures that the DefaultRunLauncher will be able to reach your run storage is here: https://github.com/dagster-io/dagster/blob/master/examples/deploy_docker/docker-compose.yml#L29-L31
j

Josh Karlin

05/12/2022, 3:19 PM
this is my own code. ah i think i sees it! those var names are different
have the expected postgres variable names changed?
d

daniel

05/12/2022, 3:20 PM
We don't assume those env vars anywhere as far as i know - whatever is put in the dagster.yaml is what it uses
j

Josh Karlin

05/12/2022, 3:20 PM
damn
d

daniel

05/12/2022, 3:20 PM
the big mystery i think is why it was working before
could the correct env vars have been set in some other way on your machine?
j

Josh Karlin

05/12/2022, 3:21 PM
i possible, i will double check. lots of config floating around
while i'm here, do you know the variable required to change the location where a dagster run writes stuff to? default is /opt/dagster which gives permission denied when running as non-root user. i will just chown it in the dockerfile but wondering if there's another way. tried DAGSTER_HOME to no avail
d

daniel

05/12/2022, 3:30 PM
when you say "writes stuff to" - what kind of stuff?
maybe worth a new post for this actually, i think we are spamming the OP a bit 🙂
👍 1
j

Josh Karlin

05/12/2022, 4:38 PM
last one here as i think this still qualifies as relevant [sorry OP] i found the source of my error - the dagster image we build for running dagit etc was missing the dagster.yaml for 0.14.12 headbang easy fix. thanks for your help Daniel. re the permissions issue, looks like at least logs:
Copy code
PermissionError: [Errno 13] Permission denied: '/opt/dagster'
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/api.py", line 775, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/usr/local/lib/python3.7/site-packages/dagster/core/executor/in_process.py", line 50, in execute
    output_capture=plan_context.output_capture,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/api.py", line 871, in __iter__
    pipeline_context=self.pipeline_context,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/execute_plan.py", line 63, in inner_plan_execution_iterator
    step_context.pipeline_run, step_context.step.key
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/compute_log_manager.py", line 55, in watch
    with self._watch_logs(pipeline_run, step_key):
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/local_compute_log_manager.py", line 49, in _watch_logs
    with mirror_stream_to_file(sys.stdout, outpath):
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/compute_logs.py", line 28, in mirror_stream_to_file
    ensure_file(filepath)
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/__init__.py", line 319, in ensure_file
    ensure_dir(os.path.dirname(path))
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/__init__.py", line 312, in ensure_dir
    os.makedirs(file_path)
  File "/usr/local/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 1 more time]
  File "/usr/local/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
d

daniel

05/12/2022, 4:39 PM
ah got it. that looks like compute logs - you can see how to configure that here: https://docs.dagster.io/deployment/dagster-instance#compute-log-storage
1000 1
j

Josh Karlin

05/12/2022, 4:41 PM
perfect, thanks again!
6 Views