https://dagster.io/ logo
p

Prratek Ramchandani

12/29/2020, 3:17 PM
Hey! Is there a reason the solids in
dagster_gcp
that load to BQ don’t support specifying schema fields in the load job config? Is it as simple as just adding the parameter here (I see a TODO)? If so, I can make a PR https://github.com/dagster-io/dagster/blob/a07fc28ae6878235aeada88e3584c6dea01f845[…]n_modules/libraries/dagster-gcp/dagster_gcp/bigquery/configs.py
s

sashank

12/29/2020, 3:20 PM
Looks like it just got stuck as TODO. A PR would be great here. cc @nate who originally wrote this
p

Prratek Ramchandani

12/29/2020, 3:21 PM
Cool I’ll do that today. Should I create an issue as well or is just a PR fine here?
n

nate

12/29/2020, 5:02 PM
just a PR is fine, thanks for taking a look at this!
👍 1
p

Prratek Ramchandani

12/29/2020, 8:00 PM
@nate quick question - would you recommend creating a custom type here to check that the schema provided is valid or should I just accept a
List[dict]
for the schema field and let the API call to BigQuery return an error if it’s an invalid schema?
Going down the custom type route since we seem to do that for the other config options.
Could one of you take a look? It’s my first contribution to an open source project so I’m not sure how to test that it works as expected - https://github.com/dagster-io/dagster/pull/3479