19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
I have another issue with Cloud that is urgent. My...
# dagster-plusk
Kevin Day
12/14/2022, 2:00 PMI have another issue with Cloud that is urgent. My scheduled jobs ran successfully this morning, but then, without any deployments on my side, my code location is totally unavailable. I have tried redeploying twice with no luck, and it looks like an internal DNS issue. Full stack trace in thread:
Kevin Day
12/14/2022, 2:01 PM Copy code
dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1098, in _reconcile
existing_pex_server_handles[to_update_key] = self._get_existing_pex_servers(
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 654, in _get_existing_pex_servers
server_endpoint.create_multipex_client()
File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 46, in get_pex_servers
res = self._query(
File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 91, in _query
raise DagsterUserCodeUnreachableError("Could not reach user code server") from e
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "DNS resolution failed for baker-prod-0e0e38.serverless-agents-namespace-2:4000: C-ares status is not ARES_SUCCESS qtype=A name=baker-prod-0e0e38.serverless-agents-namespace-2 is_balancer=0: Domain name not found" debug_error_string = "{"created":"@1671025583.021847696","description":"DNS resolution failed for baker-prod-0e0e38.serverless-agents-namespace-2:4000: C-ares status is not ARES_SUCCESS qtype=A name=baker-prod-0e0e38.serverless-agents-namespace-2 is_balancer=0: Domain name not found","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}" >
File "/dagster-cloud/dagster_cloud/pex/grpc/client.py", line 88, in _query
response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)Kevin Day
12/14/2022, 2:02 PMI have tried redeploying from Dagster and then with a new PR merge, but I still get the same issue. No definitions are loading for me.
d
daniel
12/14/2022, 2:03 PMHi Kevin - we’ll get this fixed for you shortly (within the hour). This is related to your deployment changing back from fast deploys to regular deploys, we’re rolling out a fix now
k
Kevin Day
12/14/2022, 2:03 PMPerfect. Thanks!
d
daniel
12/14/2022, 2:16 PMBelieve you should be back up and running again now - sorry for the trouble there
k
Kevin Day
12/14/2022, 2:17 PMYeah, looks good. Thanks for the fast fix!
Kevin Day
12/14/2022, 2:17 PMShould I go ahead and try out the setup.py and the fast deploys option? I'm flexible. Right now it's still using non-fast deploys.
d
daniel
12/14/2022, 2:20 PMThat should be fine - there’s just a small change we’ll need to make on our side if you switch back afterwards