Hi all, i think we need help with this error (cont...
# ask-community
s
Hi all, i think we need help with this error (context:
databricks_pyspark_step_launcher
)
Copy code
Run 962 failed with result state: DatabricksRunResultState.Failed. Message: Cannot read the python file dbfs:///dagster_staging/xxxxxxxx/pyspark_commands_1/databricks_step_main.py. Please check driver logs for more details.
and we think this is the issue in the code Images attached: i think we are adding two additional slashes and we are unable to remove the slashes from
staging_prefix.
When we try to run without the / (
"staging_prefix": "dagster_staging",
), we are getting this error
dagster._check.CheckError: Invariant failed. Description: staging_prefix must be an absolute path
. However, i think there should be only one slash after
dbfs:
. Thank you for the help https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-databricks/dagster_databricks/databricks_pyspark_step_launcher.py