The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

hey folks, another pyspark on kubernetes question. As you may know pyspark on kubernetes with multiple executor requires a headless service ( <https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode-networking|docs here> ) . I wonder if there is any out-of-box functionality to create a `headless service` that points to driver job/pod when the job is executed. Otherwise the spark executor pods can not communicate with dagster job/pod ( which is the spark driver ) .

I can add kubernetes python library and create the service in the pipeline but this requires additional authorization in kubernetes and not safe to grant dagster service account to create services.