https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
k

Kazushi Nagayama

08/09/2023, 6:43 AM
this is what i am trying out:
Copy code
@op
def create_test_view() -> None:
    bq_op_for_queries([
        "CREATE OR REPLACE VIEW `test.view`"
    ])

@op
def depends_on_test_view(creat_test_view) -> None:
    bq_op_for_queries([
        "CREATE OR REPLACE VIEW `test.another_view`"
    ])

@graph_asset
def generate_views():
    return depends_on_test_view(create_test_view())
this is the error i’m getting:
Copy code
google.api_core.exceptions.NotFound: 404 Not found: Table test:test.result was not found in location US
(i’m obfuscating the project names) i’m not querying the
result
table. what am i missing?
🤖 1
it seems like i need to define it as a job, not a graph asset
j

jamie

08/09/2023, 3:21 PM
ahhh ok so what’s happening here is that
bq_op_for_queries
actually makes the op for you (so you dont need to then wrap it in your own op. dagster is getting confusing about some of the outputs and looking for tables that don’t exist because of the double wrapping tbh i would recommend either using assets like i mentioned in the above thread, or if you want to use ops, then using the bigquery resource to execute the sql within your own op. The helper function
bq_op_for_queries
can be nice in some circumstances, but it isn’t as customizable
k

Kazushi Nagayama

08/09/2023, 3:30 PM
@jamie makes sense! yeah i ended up using bigquery resource - i'll play with what you suggested
5 Views