Hi all. I'm having some issues with writing to Del...
# ask-community
Hi all. I'm having some issues with writing to Delta tables using PySpark. My job is based on the example provided in the Dagster docs. One of the issues I'm experiencing is that Py4JError messages aren't really helpful. I will usually get something like this:
Copy code
2022-03-25 23:56:09,331 :: spark_job - e15aeb42-c6a9-4c20-b87d-f533d02029f9 - write_delta - <s3a://spark-test/delta_table_test/table_a5920b13_b182_4b5d_893c_bf2b21fc5d4c.delta>
2022-03-25 23:56:38,400 :: spark_job - e15aeb42-c6a9-4c20-b87d-f533d02029f9 - 77 - write_delta - STEP_FAILURE - Execution of step "write_delta" failed.

dagster.core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "write_delta"::

py4j.protocol.Py4JError: An error occurred while calling o186.save

Stack Trace:
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
  File "/opt/venv/lib/python3.8/site-packages/dagster/utils/__init__.py", line 396, in iterate_with_context
    next_output = next(iterator)
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/execution/plan/compute_generator.py", line 65, in _coerce_solid_compute_fn_to_iterator
    result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
  File "/usr/src/app/./pipeline/spark_test/jobs.py", line 58, in write_delta
  File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/readwriter.py", line 740, in save
  File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/opt/venv/lib/python3.8/site-packages/py4j/protocol.py", line 334, in get_return_value
    raise Py4JError(

2022-03-25 23:56:38,421 :: spark_job - e15aeb42-c6a9-4c20-b87d-f533d02029f9 - 77 - RUN_FAILURE - Execution of run for "spark_job" failed. Steps failed: ['write_delta'].
Secondly, when I try the same code in a pyspark shell, it runs without error. I'm unable to replicate most issues with Spark that I'm experiencing with Dagster. I don't doubt that there isn't an issue with my configuration, but currently I'm unable to diagnose why the same code results in an error on Dagster. Is there a way to print out the full stack trace?