I’m trying to use the dagster-gcp’s

DataprocResource

to spin up a Dataproc cluster with the component gateway enabled. However, that requires the use of the “endpointConfig” parameter in the Dataproc cluster config, which when I check in the

dagster_gcp/dataproc/configs_dataproc_cluster.py

in my virtual environment’s build of dagster-gcp, does not appear to exist. However, when I personally run the

parse_dataproc_configs.py

script (from here), the “endpointConfig” parameter (and a number of others) are in fact showing up. It seems as though maybe the

parse_dataproc_configs.py

script has not been run recently? I’m on dagster 1.2.6 and dagster-gcp 0.18.6. Any suggestions would be great, as would information about when this might be fixed or updated (or if it was intentional).

Hi Sam. The dagster-gcp library is part of the public dagster repository: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-gcp/dagster_gcp So you can file an issue within the dagster library.

@claire Thanks - I hadn’t realized that! Do you agree with the assessment that the

parse_dataproc_configs.py

looks like it just hasn’t been run in a while?

Yep, the script hasn't been run for a long time. The previous owner of this code left the company a while ago, so this library is out of date. If you'd be willing to contribute a fix for this, I'm happy to review / tag others to review. Otherwise, I'd recommend filing an issue in the meantime.

Perfect thanks! I’ll try to put together a PR, since it should just entail re-running the config parser

thanks, appreciate you contributing! you can send me a link to the PR after it's up