https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
w

won

02/25/2023, 5:15 PM
how to store df with json column or column with lists? gbq knows array ond json types. in
BigQueryPandasTypeHandler
in my case obj.dtypes returns ('items', dtype('O')),
Copy code
context.add_output_metadata(
            {
                "row_count": obj.shape[0],
                "dataframe_columns": MetadataValue.table_schema(
                    TableSchema(
                        columns=[
                            TableColumn(name=name, type=str(dtype))
                            for name, dtype in obj.dtypes.iteritems()
                        ]
                    )
                ),
            }
        )
can i use sqlalchemy.types ?
Copy code
json_columns = ['foo','baz']
data_type = dict.fromkeys(json_columns, types.JSON) if json_columns else {}
df.to_sql(
    table_name,
    connection,
    dtype=data_type,
    if_exists="append",
    index=False,
    chunksize=1000,
)
https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array-type
👍 1
4 Views