Hi, I'm trying to see if I can use dagster to mana...
# ask-community
Hi, I'm trying to see if I can use dagster to manage my spark asset dependencies. We have a Spark Cluster set up in our University's OpenShift which has a Jupyter Lab pod. I typically run my spark scripts on that pod through the command line. So, in this pod I created a new python environment and followed the instructions to install dagster. However, when I try to execute
dagster dev
I get the following error:
Copy code
2023-08-14 07:38:06 +0000 - dagster - INFO - Using temporary directory /home/jovyan/work/dagster_test/my-dagster-project/tmpezu95xhw for storage. This will be removed when dagster dev exits.
2023-08-14 07:38:06 +0000 - dagster - INFO - To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.
Bus error (core dumped)
I think this might be a permission issue in the OpenShift container/pod. What permissions and resources does dagster require? Thanks in advance for the help.
one relevant bit from those logs is with out the
env var set, dagster creates a temporary directory in the current working directory, its possible that directory is not ok to use in that environment and you need to set the env var to point at a safe directory to write to
Hi @alex thanks for the suggestion. I created a temp folder and set the
variable. When running
dagster dev
it still crashes with the same
Bus Error (core dumped)
Another colleague was able to debug a bit further and said this
SIGBUS is happening in Dagster working with its SQLite database.
We think it might be some issue with the SQLlite version we have on this pod as dagster seems to work on another container we have running.