https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Akira Chang

04/18/2023, 1:10 PM
Hello! Can anyone help me with the api
bq_op_for_queries
for querying through BigQuery? I couldn't find any documentation about it, would I get the result of my query directly if I use this API?
t

Tim Castillo

04/18/2023, 1:13 PM
a

Akira Chang

04/18/2023, 1:26 PM
Thank you for your reply! My code looks like this
Copy code
@asset 
def an_asset():
 df = bq_op_for_queries("**my queries**")
 print(df)
I just want to simply print out the results of my custom query, I get a type
OpDefinition
as the result, why is that?
t

Tim Castillo

04/18/2023, 1:36 PM
Aaah, it's because
bq_op_for_queries
makes an Op and isn't made to be used within an asset code. If you're familiar with Airflow, it's like the
BigQueryOperator
. I remember talking to you about the
bigquery_resource
and using that to run queries. Did that not fit your needs?
a

Akira Chang

04/18/2023, 1:44 PM
ahh I see I see. Yes, it did, but I thought it was abit of work to add my asset eveytime to the
Definition
object when I needed to run queries on my
BigQuery tables
So I thought if
bq_op_for_queries
would be any simpler. I was also thinking of using class-based resources for querying my tables, it seems that they will be releasing an evolution to the
bigquery_resource
this week?
t

Tim Castillo

04/18/2023, 1:45 PM
Hmm. How are you adding your assets to your
Definitions
object? In most situations, if you're using
load_assets_from_module
, you'd only have to explicitly update your assets in the
Definitions
if you made a new Python module.
a

Akira Chang

04/18/2023, 1:50 PM
So for example I have project like
Copy code
-My_project
 -Pipeline1 Folder
  - assets.py
 -Pipeline2 Folder
  -assets.py
 -Pipeline3 Folder
  -assets.py
 -__init__.py
like this, and I have three
assets.py
file in three different folders(Pipeline1,2,3), the
Definitions
object is defined in the
init
file. Can I still use
load_assets_from_module
to load all of the assets from all three pipeline folders to the
Definitions
object?
t

Tim Castillo

04/18/2023, 1:52 PM
Yeup! It'd be 3 calls total (1 per folder), and then you can add them together in your
Definitions
object. Here's an example of how 3 different modules had their assets loaded and the asset groups were all combined together into one
Definitions
🔥 1
Hmm, this is good feedback though. We don't do this in the tutorial and we probably should
🙏 1
a

Akira Chang

04/18/2023, 1:53 PM
Ahhhh interesting I see! I will try it out! Thank you again for your help 🔥🙏
When I run
dagster dev
in
Pipeline_1
I get this error
Copy code
UserWarning: Error loading repository location Pipeline_1:dagster._core.errors.DagsterInvariantViolationError: No repositories, jobs, pipelines, graphs, asset groups, or asset definitions found in "Pipeline_1".
I've put the Definitions object in the
init.py
file under
My_project
like we mentioned above
My pipelines are different
scaffolds
, sorry I didnt mention that earlier. Does it still work in this case?
t

Tim Castillo

04/18/2023, 3:51 PM
you'll likely have to run
dagster dev
from your
My_project
directory.