hello, i'm trying to figure out how to use dagster-pyspark. i run spark on k8s so i have to set a lot of spark config variables (k8s serviceaccount, ...) i try pyspark_resource.configured({"spark_conf":{}}) but it seems to ignore a lot of my variables, eg spark.kubernetes.authenticate.driver.serviceAccountName

Cc @bob @johann

@Frank Dekervel currently, dagster doesn’t recognize k8s config that’s specified in

spark_conf

. that’s why your

spark.kubernetes.*

variables are being ignored alternatively, a solution would be to specify user-defined k8s config on your solid or pipeline that uses the

pyspark_resource

. this is done with the

tags

argument in the

@solid

and

@pipeline

decorator i made a quick example below for specifying the k8s ServiceAccountName:

@pipeline(
    # ...other parameters,
    tags={
        'dagster-k8s/config': {
            'pod_spec_config': {
                'serviceAccountName': 'my-k8s-service-account',
            },
        },
    },
)
def my_pipeline_that_uses_pyspark():
    pass

if you have additional variables that usually go in

spark.kubernetes.*

, you might be able to fit them in

tags['dagster-k8s/config]

another example can be found in the changelog announcement of “user-defined k8s config”

is there a reason why it doesn't simply pass through all spark_conf variables ? because i can imagine that it will be hard to be complete here ... other users might want to set spark.hadoop.* variables eg for S3 access too

i see. for reference, here is the Python script that generates the schema of “spark_conf”. it lists out the config variables that would be recognized

so in order to support the spark k8s vars they would need to be added to CONFIG_TYPES ?

regarding your question about adding spark k8s vars to

CONFIG_TYPES

, that wouldnt be sufficient 😓

parse_spark_configs.py

fetches the markdown for the Spark config doc website and generates the Dagster config*,* but the config variables for Spark kubernetes stuff is on a different page :///

CONFIG_TYPES

is there for typing non-string config variables. by default, all the Spark config variables will be

str

typed

ok, fetching and parsing another page wouldn't be too hard ... but that doesn't solve the problem with k8s annotations

altho, many of the fields in

CONFIG_TYPES

are still

ConfigType.String

🤔 hmm

cc @sandy

yeah, it looks like the key/values of

spark_config

end up being passed to `SparkSession.builder` which should recognize annotations or whatnot

hey @Frank Dekervel - if you do the following, "spark.some.config" should end up in the configuration for the SparkSession instantiated by the pyspark resource, even if "spark.some.config" isn't on the list we scraped from the spark website

pyspark_resource.configured({"spark_conf":{"spark.some.config": "some_value"}})

if you generate a SparkSession on your own with that config, do you encounter that same error?

well i use the same set of variables on spark-submit and with polynote and there it works

using the pyspark resource should be exactly equivalent to instantiating a SparkSession inside your solid and providing it with the configuration you're describing

yeah its making it in the spark_conf .. i just did resource._configured_config_mapping_fn({}) and i still see the serviceaccount config there in the result

spark + k8s is a very complex beast

yeah ... after fixing this spark lauches the worker nodes ... which fail to connect to the driver node, so that's my next problem

glad you're making progress!

ok, i have the EMR example now working on an on-premises openshift cluster