https://dagster.io/ logo
Join Slack
Powered by
Error with spark on windows `Java gateway process ...
# announcements
s
Error with spark on windows 
Java gateway process exited before sending its port number
(full stack trace see in first comment). It works fine on my Macbook and linux, but in my a bit more complicated work environment with windows and WSL plus proxy, my local spark connection gives the above error. I tried several things with JAVA options for proxy etc. but before I search more, do you might have an idea where I should start searching or how to fix that? Would be much appreciated.
Copy code
Exception: Java gateway process exited before sending its port number
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/dagster/core/errors.py", line 174, in user_code_error_boundary
    yield
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/dagster/core/execution/resources_init.py", line 148, in single_resource_event_generator
    resource = next(gen)
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/dagster_pyspark/resources.py", line 59, in pyspark_resource
    pyspark = SystemPySparkResource(init_context.resource_config['spark_conf'])
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/dagster_pyspark/resources.py", line 23, in __init__
    self._spark_session = spark_session_from_config(spark_conf)
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/dagster_pyspark/resources.py", line 18, in spark_session_from_config
    return builder.getOrCreate()
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/pyspark/sql/session.py", line 173, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/pyspark/context.py", line 367, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/pyspark/context.py", line 133, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/pyspark/context.py", line 316, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/pyspark/java_gateway.py", line 46, in launch_gateway
    return _launch_gateway(conf)
  File "/home/spaeti/.venvs/dagster/lib/python3.6/site-packages/pyspark/java_gateway.py", line 108, in _launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
pyspark as such works fine. You can also see my additional JAVA-settings
s
You might have come across this already, but this thread seems to have a few different solutions: https://github.com/jupyter/jupyter/issues/248 A lot of people seem to be reporting fixing it by setting 
JAVA_HOME
, if you haven't tried that already
Open in Slack
PreviousNext
Powered by