Hello! I am trying to gather metadata about dbt runs in bigquery. I am attempting to use the

runtime_metadata_fn

to gather this metadata about partitioned builds. The custom function I am using and the

load_assets_from_dbt_project

read as follows:

def custom_metadata_fn(context, node_info):
    table_name = node_info["name"]
    table_schema = node_info["schema"]
    n_rows = f"SELECT 1 from {table_schema}.{table_name} where date = {context.partition_key}"
    return {"n_rows": n_rows}

dbt_assets = load_assets_from_dbt_project(
    project_dir=DBT_PROJECT_PATH,
    profiles_dir=DBT_PROFILES,
    use_build_command=True,
    partitions_def=DailyPartitionsDefinition(start_date="2023-04-14"),
    partition_key_to_vars_fn=partition_key_to_dbt_vars,
    runtime_metadata_fn=custom_metadata_fn,
)

The question I have is how do I pass that

n_rows

query to either an io_manager or some other mechanism to actually query the

{table_schema}.{table_name}

table? do i just have to access the bigquery sdk? or is there a dagster api I can interface with? using:

dbt-bigquery==1.1.0
dagster==1.3.0
dagit==1.3.0
dagster-dbt==0.19.0

Hey @Riley Runnoe! So an intricacy of the dbt integration is that it doesn’t use the Dagster IO manager to store the tables in the DB, it uses the underlying dbt functionality. Only when you load the asset in a downstream asset does the Dagster IO manager kick in to fetch the table. You definitely could use the bigquery sdk to manually create a client and run the command yourself. @owen do you have any other ideas for how to run a sql query like that to get the metadata?