https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
a

Andrew Vaccaro

08/09/2023, 12:04 AM
Hey, I’m testing out dagster as an alternative for airflow in my typical stack, and I’m wondering if there’s a typical pattern for using BigQuery external tables with data stored in GCS (often in a hive-partitioned key structure). We’ve used that pattern on https://github.com/cal-itp to pretty good success; it allows us to “pull” data into bigquery via dbt incremental materializations. (Asking because I’m currently attempting to have the io manager use hive-partitioned keys)
🤖 1
y

yuhan

08/09/2023, 7:04 AM
here’s a guide to bigquery w/ dagster: https://docs.dagster.io/integrations/bigquery/reference
if you are dealing with bigquery directly (without dbt), this guide would help^ but if you’d like to pull data into bigquery via dbt, you may check out the guides here: https://docs.dagster.io/integrations/dbt — when using dbt, it becomes dbt that manages/connects to your bigquery, so you may want to specify configuration via dbt profile than dagster io managers
🙏 1
2 Views