https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
m

Miguel Caballero Pinto

05/23/2023, 12:08 AM
Hi there! Im trying to output the result of a software-defined asset to an existing BigQuery table. I use the
BigQueryPandasIOManager
. When the job is going to write to the table it fails because one of the columns in the dataframe is being translated to integer instead of string. This seems to happen because the column in the dataframe is None and dagster by default may infer that the type is an integer. This is the error I get:
Copy code
google.api_core.exceptions.BadRequest: 400 Provided Schema does not match Table innate-empire-283902:enrichment.person_batch_enrichment_output. Field error_code has changed type from STRING to INTEGER
Is there a way to workaround this? How can I tell dagster that the output type of the column is a string?
o

owen

05/23/2023, 12:17 AM
Hi @Miguel Caballero Pinto! The bigquery connector that Dagster is using under the hood is just using the type information that's available on the pandas DataFrame object in order to predict what type a column should be translated to. It sounds like you're correct that this is related to the column being None and this guess being incorrect. One option here would be to manually convert the type of your
error_code
column within pandas, i.e.
Copy code
my_df = ...
my_df.error_code = my_df.error_code.astype(str)
this should ensure that this column is interpreted as a string
m

Miguel Caballero Pinto

05/23/2023, 12:18 AM
Ok! Or maybe specify the
dtype
argument of the dataframe constructor?
o

owen

05/23/2023, 12:26 AM
yep! that'd work as well
9 Views