hey, i'm looking to make a start on <https://githu...
# announcements
b
hey, i'm looking to make a start on https://github.com/dagster-io/dagster/issues/2458 and basing it off the
dagster_aws.emr
subpackage. i've started with the types module, but it looks like the
dagster_aws.emr.types
module uses both regular python enums and dagster enums - why is that? have i missed something in the docs?
oh, it looks like dagster enums are used for inputs and pyenums everywhere else
s
hey Ben - are you planning to launch Databricks clusters as part of your jobs, or submit to existing Databricks clusters?
I believe emr/types mostly concerns configuring cluster launch, so if you're focused on the latter, you might not need to build an equivalent to it (the emr_pyspark_step_launcher doesn't currently rely on those types)
b
it'll probably be to new clusters to be honest; databricks' pricing is based on whether you submit jobs to a new or existing cluster (https://docs.databricks.com/dev-tools/api/latest/jobs.html#request-structure)
the types are fairly well defined by the API docs at least so it should be fairly simple to translate to dagster types
s
"We suggest running jobs on new clusters for greater reliability." - interesting
b
yeah it's a bit counterintuitive but they at least have a single API endpoint (Runs Submit) to run a job on a new cluster, so it might be simpler than the EMR launcher anyways
s
that sounds much simpler