https://dagster.io/ logo
#dagster-cloud
Title
# dagster-cloud
k

Kevin Day

12/14/2022, 2:00 PM
I have another issue with Cloud that is urgent. My scheduled jobs ran successfully this morning, but then, without any deployments on my side, my code location is totally unavailable. I have tried redeploying twice with no luck, and it looks like an internal DNS issue. Full stack trace in thread:
Copy code
dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1098, in _reconcile
    existing_pex_server_handles[to_update_key] = self._get_existing_pex_servers(
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 654, in _get_existing_pex_servers
    server_endpoint.create_multipex_client()
  File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 46, in get_pex_servers
    res = self._query(
  File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 91, in _query
    raise DagsterUserCodeUnreachableError("Could not reach user code server") from e
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "DNS resolution failed for baker-prod-0e0e38.serverless-agents-namespace-2:4000: C-ares status is not ARES_SUCCESS qtype=A name=baker-prod-0e0e38.serverless-agents-namespace-2 is_balancer=0: Domain name not found" debug_error_string = "{"created":"@1671025583.021847696","description":"DNS resolution failed for baker-prod-0e0e38.serverless-agents-namespace-2:4000: C-ares status is not ARES_SUCCESS qtype=A name=baker-prod-0e0e38.serverless-agents-namespace-2 is_balancer=0: Domain name not found","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}" >
  File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 88, in _query
    response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
I have tried redeploying from Dagster and then with a new PR merge, but I still get the same issue. No definitions are loading for me.
d

daniel

12/14/2022, 2:03 PM
Hi Kevin - we’ll get this fixed for you shortly (within the hour). This is related to your deployment changing back from fast deploys to regular deploys, we’re rolling out a fix now
k

Kevin Day

12/14/2022, 2:03 PM
Perfect. Thanks!
d

daniel

12/14/2022, 2:16 PM
Believe you should be back up and running again now - sorry for the trouble there
k

Kevin Day

12/14/2022, 2:17 PM
Yeah, looks good. Thanks for the fast fix!
Should I go ahead and try out the setup.py and the fast deploys option? I'm flexible. Right now it's still using non-fast deploys.
d

daniel

12/14/2022, 2:20 PM
That should be fine - there’s just a small change we’ll need to make on our side if you switch back afterwards