Hey team, I am running into the following error when scheduling ~400 pipelines in Dagster at once (with a max queue size of 20 runs in parallel). I am using v0.11.13. Any ideas?

PIPELINE_FAILURE
Caught an error for run a564083e-8f75-45a9-b9a2-63e5382d5e52 while removing it from the queue. Marking the run as failed and dropping it from the queue: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses"
	debug_error_string = "{"created":"@1631520273.921130387","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3008,"referenced_errors":[{"created":"@1631520273.921127399","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"
>

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py", line 154, in run_iteration
    self._dequeue_run(instance, run, workspace)
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/run_coordinator/queued_run_coordinator_daemon.py", line 206, in _dequeue_run
    location = workspace.get_location(repository_location_origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/workspace/dynamic_workspace.py", line 36, in get_location
    location = existing_location if existing_location else origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/origin.py", line 255, in create_location
    return GrpcServerRepositoryLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 478, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.7/site-packages/dagster/api/list_repositories.py", line 13, in sync_list_repositories_grpc
    api_client.list_repositories(), (ListRepositoriesResponse, SerializableErrorInfo)
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 143, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 89, in _query
    response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)

Hi Ruben - as of 0.12.0 the run dequeuer no longer reaches out to the gRPC server, so if upgrading is an option I’d expect the error in this particular callsite at least to go away