hey folks another pyspark on kubernetes question As you may know pyspark on kubernetes with multiple executor requires a headless service <https spark apache org docs latest running on kubernetes html client mode networking|docs here> I wonder if there is any out of box functionality to create a `headless service` that points to driver job pod when the job is executed Otherwise the spark executor pods can not communicate with dagster job pod which is the spark driver

hey folks another pyspark on kubernetes question As you may dagster #dagster-support

Title

# dagster-support

Baris Cekic

09/11/2022, 10:57 AM

hey folks, another pyspark on kubernetes question. As you may know pyspark on kubernetes with multiple executor requires a headless service ( docs here ) . I wonder if there is any out-of-box functionality to create a

headless service

that points to driver job/pod when the job is executed. Otherwise the spark executor pods can not communicate with dagster job/pod ( which is the spark driver ) .

sean

09/14/2022, 8:40 PM

cc @daniel @rex

Baris Cekic

09/21/2022, 10:26 PM

any update guys ?

I can add kubernetes python library and create the service in the pipeline but this requires additional authorization in kubernetes and not safe to grant dagster service account to create services.