Hi, I'm trying to see if I can use dagster to manage my spark asset dependencies. We have a Spark Cluster set up in our University's OpenShift which has a Jupyter Lab pod. I typically run my spark scripts on that pod through the command line. So, in this pod I created a new python environment and followed the instructions to install dagster. However, when I try to execute

dagster dev

I get the following error:

2023-08-14 07:38:06 +0000 - dagster - INFO - Using temporary directory /home/jovyan/work/dagster_test/my-dagster-project/tmpezu95xhw for storage. This will be removed when dagster dev exits.
2023-08-14 07:38:06 +0000 - dagster - INFO - To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.
Bus error (core dumped)

I think this might be a permission issue in the OpenShift container/pod. What permissions and resources does dagster require? Thanks in advance for the help.

one relevant bit from those logs is with out the

DAGSTER_HOME

env var set, dagster creates a temporary directory in the current working directory, its possible that directory is not ok to use in that environment and you need to set the env var to point at a safe directory to write to

Hi @alex thanks for the suggestion. I created a temp folder and set the

DAGSTER_HOME

variable. When running

dagster dev

it still crashes with the same

Bus Error (core dumped)