https://dagster.io/ logo
Title
d

Dennis Hendricks

05/22/2023, 2:33 PM
Hey everyone! I'm trying to create a partitioned table in BigQuery via
BigQueryPandasIOManager
. The partition in Dagster itself works (using
DailyPartitionsDefinition
), but the result in BigQuery is a regular table instead of one with the table type
Partitioned
. Is there a way to specify the table type directly via Dagster or is my only option to create the table manually in BigQuery and populate it via Dagster?
j

jamie

05/22/2023, 2:36 PM
Hey @Dennis Hendricks this BQ IO Manager doesn’t create Partitioned type tables right now (we manually determine which rows to replace when partitions are re-materialized). I’d like to add native BQ partition support at some point, but it’s not a priority on our roadmap right now
d

Dennis Hendricks

05/22/2023, 2:37 PM
Allright, thanks for the quick feedback @jamie 🙌
j

jamie

05/22/2023, 2:39 PM
yep! if it’s an important feature for you or something you would be interested in contributing, I’m happy to help out and explain how the BQ io manager works