Hi all. I'm having some issues with writing to Delta tables using PySpark. My job is based on the example provided in the Dagster docs. One of the issues I'm experiencing is that Py4JError messages aren't really helpful. I will usually get something like this:

2022-03-25 23:56:09,331 :: spark_job - e15aeb42-c6a9-4c20-b87d-f533d02029f9 - write_delta - <s3a://spark-test/delta_table_test/table_a5920b13_b182_4b5d_893c_bf2b21fc5d4c.delta>
2022-03-25 23:56:38,400 :: spark_job - e15aeb42-c6a9-4c20-b87d-f533d02029f9 - 77 - write_delta - STEP_FAILURE - Execution of step "write_delta" failed.

dagster.core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "write_delta"::

py4j.protocol.Py4JError: An error occurred while calling o186.save

Stack Trace:
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
    yield
  File "/opt/venv/lib/python3.8/site-packages/dagster/utils/__init__.py", line 396, in iterate_with_context
    next_output = next(iterator)
  File "/opt/venv/lib/python3.8/site-packages/dagster/core/execution/plan/compute_generator.py", line 65, in _coerce_solid_compute_fn_to_iterator
    result = fn(context, **kwargs) if context_arg_provided else fn(**kwargs)
  File "/usr/src/app/./pipeline/spark_test/jobs.py", line 58, in write_delta
    df.write.format("delta").mode("overwrite").save(table_path)
  File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/readwriter.py", line 740, in save
    self._jwrite.save(path)
  File "/opt/venv/lib/python3.8/site-packages/py4j/java_gateway.py", line 1321, in __call__
    return_value = get_return_value(
  File "/opt/venv/lib/python3.8/site-packages/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/opt/venv/lib/python3.8/site-packages/py4j/protocol.py", line 334, in get_return_value
    raise Py4JError(

2022-03-25 23:56:38,421 :: spark_job - e15aeb42-c6a9-4c20-b87d-f533d02029f9 - 77 - RUN_FAILURE - Execution of run for "spark_job" failed. Steps failed: ['write_delta'].

Secondly, when I try the same code in a pyspark shell, it runs without error. I'm unable to replicate most issues with Spark that I'm experiencing with Dagster. I don't doubt that there isn't an issue with my configuration, but currently I'm unable to diagnose why the same code results in an error on Dagster. Is there a way to print out the full stack trace?