Hi everyone Sometimes I get a Java connection refused error

Question

Hi everyone Sometimes I get a Java connection refused error `ERROR root Exception while sending command ` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1200 in send command` `answer = smart decode self stream readline 1 ` `File usr local lib python3 8 socket py line 669 in readinto` `return self sock recv into b ` `ConnectionResetError Errno 104 Connection reset by peer` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1033 in send command` `response = connection send command command ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1211 in send command` `raise Py4JNetworkError ` `py4j protocol Py4JNetworkError Error while receiving` `ERROR root Exception while sending command ` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1207 in send command` `raise Py4JNetworkError Answer from Java side is empty ` `py4j protocol Py4JNetworkError Answer from Java side is empty` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1033 in send command` `response = connection send command command ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1211 in send command` `raise Py4JNetworkError ` `py4j protocol Py4JNetworkError Error while receiving` `ERROR py4j java gateway An error occurred while trying to connect to the Java server 127 0 0 1 41979 ` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 977 in get connection` `connection = self deque pop ` `IndexError pop from an empty deque` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1115 in start` `self socket connect self address self port ` `ConnectionRefusedError Errno 111 Connection refused` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 977 in get connection` `connection = self deque pop ` `IndexError pop from an empty deque` `During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1115 in start` `self socket connect self address self port ` `ConnectionRefusedError Errno 111 Connection refused`

David Nogueira · Answer

`During handling of the above exception another exception occurred ` `Traceback most recent call last ` `File tmp tmpeznsler9 py line 127 in main` `list run step from ref step run ref instance ` `File databricks python lib python3 8 site packages dagster core execution plan external step py line 279 in run step from ref` `step context = step run ref to step context step run ref instance ` `File databricks python lib python3 8 site packages dagster core execution plan external step py line 238 in step run ref to step context` `pipeline = step run ref recon pipeline subset for execution from existing pipeline ` `File databricks python lib python3 8 site packages dagster core definitions reconstruct py line 230 in subset for execution from existing pipeline` `return self subset for execution solids to execute=solids to execute solid selection=None ` `File databricks python lib python3 8 site packages dagster core definitions reconstruct py line 183 in subset for execution` `pipeline def = self get definition ` `File databricks python lib python3 8 site packages dagster core definitions reconstruct py line 152 in get definition` `defn = self repository get definition get pipeline self pipeline name ` `File databricks python lib python3 8 site packages dagster core definitions reconstruct py line 74 in get definition` `return repository def from pointer self pointer ` `File databricks python lib python3 8 site packages dagster core definitions reconstruct py line 628 in repository def from pointer` `target = def from pointer pointer ` `File databricks python lib python3 8 site packages dagster core definitions reconstruct py line 549 in def from pointer` `target = pointer load target ` `File databricks python lib python3 8 site packages dagster core code pointer py line 201 in load target` `module = load python module self module self working directory ` `File databricks python lib python3 8 site packages dagster core code pointer py line 125 in load python module` `return importlib import module module name ` `File usr local lib python3 8 importlib init py line 127 in import module` `return bootstrap gcd import name level package level ` `File lt frozen importlib bootstrap gt line 1014 in gcd import` `File lt frozen importlib bootstrap gt line 991 in find and load` `File lt frozen importlib bootstrap gt line 975 in find and load unlocked` `File lt frozen importlib bootstrap gt line 671 in load unlocked` `File lt frozen importlib bootstrap external gt line 783 in exec module` `File lt frozen importlib bootstrap gt line 219 in call with frames removed` `File tmp tmp9 012ifp cashback spark tests utilities results assertion py line 1 in lt module gt ` `from cashback spark utilities pyspark dataframes import flatten df` `File databricks python shell dbruntime PythonPackageImportsInstrumentation init py line 158 in import patch` `original result = python builtin import name globals locals fromlist level ` `File tmp tmp9 012ifp cashback spark init py line 1 in lt module gt ` `from repository import cashback spark` `File databricks python shell dbruntime PythonPackageImportsInstrumentation init py line 158 in import patch` `original result = python builtin import name globals locals fromlist level ` `File tmp tmp9 012ifp cashback spark repository py line 40 in lt module gt ` `def cashback spark ` `File databricks python lib python3 8 site packages dagster core definitions decorators repository decorator py line 238 in repository` `return Repository name ` `File databricks python lib python3 8 site packages dagster core definitions decorators repository decorator py line 33 in call ` `repository definitions = fn ` `File tmp tmp9 012ifp cashback spark repository py line 41 in cashback spark` `jobs = load jobs ` `File tmp tmp9 012ifp cashback spark repository py line 27 in load jobs` `return loader folder= jobs ` `File tmp tmp9 012ifp cashback spark repository py line 20 in loader` `module = import module module name ` `File usr local lib python3 8 importlib init py line 127 in import module` `return bootstrap gcd import name level package level ` `File tmp tmp9 012ifp cashback spark jobs addresses job py line 1 in lt module gt ` `from cashback spark ops addresses import add region get coordinates ` `File databricks python shell dbruntime PythonPackageImportsInstrumentation init py line 158 in import patch` `original result = python builtin import name globals locals fromlist level ` `File tmp tmp9 012ifp cashback spark ops addresses py line 131 in lt module gt ` `def compare2continent lat lng ` `File databricks spark python pyspark databricks sql functions py line 72 in create edge udf` `session conf get ` `File databricks spark python pyspark sql conf py line 45 in get` `return self jconf get key ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1303 in call ` `answer = self gateway client send command command ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1031 in send command` `connection = self get connection ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 979 in get connection` `connection = self create connection ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 985 in create connection` `connection start ` `File databricks spark python lib py4j 0 10 9 src zip py4j java gateway py line 1127 in start` `raise Py4JNetworkError msg e ` `py4j protocol Py4JNetworkError An error occurred while trying to connect to the Java server 127 0 0 1 41979 ` `2022 09 08 12 22 10 +0100 dagster ERROR assert wrapper c769ccf6 3823 4846 ac98 b72de0feaf23 7605 assert tables STEP FAILURE Execution of step assert tables failed ` `dagster databricks databricks DatabricksError Run 1747310 failed with result state DatabricksRunResultState Failed Message ` `Stack Trace ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution plan execute plan py line 232 in dagster event sequence for step` `for step event in check generator step events ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks pyspark step launcher py line 181 in launch step` `yield from self step events iterator step context step key databricks run id ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks pyspark step launcher py line 223 in step events iterator` `done = poll run state ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks py line 272 in poll run state` `raise DatabricksError error message ` `2022 09 08 12 22 10 +0100 dagster ERROR assert wrapper c769ccf6 3823 4846 ac98 b72de0feaf23 7605 RUN FAILURE Execution of run for assert wrapper failed An exception was thrown during execution `

David Nogueira · Answer

`dagster databricks databricks DatabricksError Run 1747310 failed with result state DatabricksRunResultState Failed Message ` `Stack Trace ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 785 in pipeline execution iterator` `for event in pipeline context executor execute pipeline context execution plan ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core executor in process py line 38 in execute` `yield from iter ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 879 in iter ` `yield from self iterator ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution plan execute plan py line 87 in inner plan execution iterator` `for step event in check generator dagster event sequence for step step context ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution plan execute plan py line 336 in dagster event sequence for step` `raise unexpected exception` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution plan execute plan py line 232 in dagster event sequence for step` `for step event in check generator step events ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks pyspark step launcher py line 181 in launch step` `yield from self step events iterator step context step key databricks run id ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks pyspark step launcher py line 223 in step events iterator` `done = poll run state ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks py line 272 in poll run state` `raise DatabricksError error message ` `Traceback most recent call last ` `File mnt c Users david Repos D4DZarco cashback spark cashback spark tests test jobs subpayments job py line 45 in lt module gt ` `execute pipeline reconstructable assert wrapper run config=config mode= prod adls2 ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 383 in execute pipeline` `return logged execute pipeline ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core telemetry py line 110 in wrap` `result = f args kwargs ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 434 in logged execute pipeline` `return execute run ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 236 in execute run` `event list = list execute run iterable ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 879 in iter ` `yield from self iterator ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 785 in pipeline execution iterator` `for event in pipeline context executor execute pipeline context execution plan ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core executor in process py line 38 in execute` `yield from iter ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution api py line 879 in iter ` `yield from self iterator ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution plan execute plan py line 87 in inner plan execution iterator` `for step event in check generator dagster event sequence for step step context ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution plan execute plan py line 336 in dagster event sequence for step` `raise unexpected exception` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster core execution plan execute plan py line 232 in dagster event sequence for step` `for step event in check generator step events ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks pyspark step launcher py line 181 in launch step` `yield from self step events iterator step context step key databricks run id ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks pyspark step launcher py line 223 in step events iterator` `done = poll run state ` `File home dan anaconda3 envs dagster 0 14 6 lib python3 9 site packages dagster databricks databricks py line 272 in poll run state` `raise DatabricksError error message ` `dagster databricks databricks DatabricksError Run 1747310 failed with result state DatabricksRunResultState Failed Message `

David Nogueira · Answer

Any hint

yuhan · Answer

the error seems to be surfaced from databricks side cc < owen> do you have any hint for troubleshooting

owen · Answer

hard to tell from the stack trace have you been able to successfully use the step launcher before and this error just started showing up

owen · Answer

also do you have access to the databricks console if so the driver logs might have more detailed stdout stderr info

David Nogueira · Answer

Thanks I ll be waiting for next time it occurs to check the databricks logs as it only shows the last few hours