Hey everyone,
is the following local setup possible:
Have Dagster run pyspark job using the spark-on-k8s-operator from google in a Minikube cluster on a schedule?
#mydreamsetup
Or simply running a pyspark job on minikube?
I am looking for a beginner walkthrough of some sort (spark + k8s +dagster)
s
sandy
02/23/2022, 5:02 PM
Hi @timo nicolas - I believe it should be possible - at the very least, you should be able to use
dagster-shell
to invoke it from the command line. However, I don't have familiarity with this particular setup