Sai Gopinath
03/07/2023, 11:22 PMdatabricks_pyspark_step_launcher
)
Run 962 failed with result state: DatabricksRunResultState.Failed. Message: Cannot read the python file dbfs:///dagster_staging/xxxxxxxx/pyspark_commands_1/databricks_step_main.py. Please check driver logs for more details.
and we think this is the issue in the code
Images attached: i think we are adding two additional slashes and we are unable to remove the slashes from staging_prefix.
When we try to run without the / ("staging_prefix": "dagster_staging",
), we are getting this error dagster._check.CheckError: Invariant failed. Description: staging_prefix must be an absolute path
. However, i think there should be only one slash after dbfs:
.
Thank you for the help
https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-databricks/dagster_databricks/databricks_pyspark_step_launcher.py