https://dagster.io/ logo
#announcements
Title
# announcements
f

Flavien

05/28/2020, 1:04 PM
Hello! First of all, congrats for this fantastic tool that is Dagster. I have one little question though: the
dagster_spark
package doesn't seem to have configurations to support Hive (
spark.sql.warehouse.dir
and
.enableHiveSupport
). Will it be possible to do so in an upcoming release ? If not, are you open to PR ?
s

sandy

05/28/2020, 3:10 PM
Hi Flavien - you're right that this is a gap for us. I can take this on for our upcoming release. Of course, if you're interested in contributing, that would be great too.
Actually, I dug a little more, and you should be able to pass these configurations already. The spark config is "permissive", so you can pass configs that aren't explicitly defined e.g. this should work:
Copy code
resources:
  pyspark:
    config:
      spark_conf:
        spark.sql.warehouse.dir: some_dir
have you tried something similar and run into errors?
3 Views