trying to spin up dagster examples with docker (used example listed here) for local testing my code repo with pipelines and sensors however getting error while starting grpc server. tried adding configs in dagster.yaml but looks like its not picked up as default grpc timeout is still 60 seconds

code_servers:
  local_startup_timeout: 3600

dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py", line 588, in _load_location
    location = self._create_location_from_origin(origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py", line 508, in _create_location_from_origin
    return origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/origin.py", line 325, in create_location
    return GrpcServerCodeLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/code_location.py", line 598, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.7/site-packages/dagster/_api/list_repositories.py", line 19, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 231, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 160, in _query
    e, timeout=timeout, custom_timeout_message=custom_timeout_message
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 144, in _raise_grpc_exception
    ) from e
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses" debug_error_string = "{"created":"@1680185966.771586903","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1680185966.771586144","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}" >
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 157, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 132, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)

Hi Gurjit - if the grpc server is running in a separate Docker container it isn't considered 'local' for the purpose of that timeout. If the problem is actually that your code takes more than 60 seconds to load, you may need to set up the docker-compose file so that dagit waits for the user code container to finish loading before it starts up - or reload things from the Code Locations tab in dagit once your code is finished loading

tried reloading from dagit but still getting same grpc error as looks like grpc server is not able to load repo in 60 seconds. will try to see if i can configure docker compose to wait for grpc but problem is grpc timeout is 60 seconds by default which i am not able to modify (below is command from exmaples directory)

CMD ["dagster", "api", "grpc", "--lazy-load-user-code", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"]

Once the grpc server has loaded its code, you should be able to reload the code location from dagit (after any amount of time - the 60 second timeout shouldn't matter here)

tried couple times but dagit doesn't load the code and shows same timeout error. verified grpc is running and listening

What if you reload the code location in Dagit?

getting grpc timeout error even though grpc server is running -

That's using the same code in the example here that you posted? https://github.com/dagster-io/dagster/tree/master/examples/deploy_docker

nope. have added more jobs/jobs and sensors to test out on my local.. the sample code provide above in github loads fine as its just couple ops..

Any chance you can share the code?

nope. won't be able to share it as its not public and customized to organization. can you suggest some configs for dagit to increase grpc timeout ..

Can you give a rough sense of the number of jobs and sensors so that we can attempt to reproduce the problem? There's a DAGSTER_GRPC_TIMEOUT_SECONDS env var that you can set to increase the timeout

has around 1000 jobs. let me try that config for dagit..