Hello I m making a `SourceAsset` that connects to my BigQuer dagster #ask-community

Hello, I'm making a `SourceAsset` that connects to...

Akira Chang

04/11/2023, 9:55 AM

Hello, I'm making a

SourceAsset

that connects to my BigQuery database. I see online that there is a lib that I can use to connect. But can I run query on my BigQuery tables with this library as well?

Tim Castillo

04/11/2023, 2:47 PM

Hi! The link you posted is a guide on using BigQuery as an I/O Manager, a Dagster concept. You can also run your own queries using the more high-level BigQuery resource. Here are some docs on it: https://docs.dagster.io/integrations/bigquery/reference#executing-custom-sql-commands-with-the-bigquery-resource

Akira Chang

04/17/2023, 9:21 AM

Hi! Thank you so much for your reply!

Akira Chang

04/17/2023, 1:59 PM

So can I use

bigquery_pandas_io_manager

to query as well? Because according to the doc, only

bigquery_resource

covers the custom query.

Akira Chang

04/17/2023, 2:56 PM

I get this error when I query through my table with the method provided from the doc

Copy code

dagster._core.errors.DagsterInvariantViolationError: Object <google.cloud.bigquery.table.RowIterator object at 0x7fdda86779dC> is not picklable. You are currently using the fs_io_manager and the multi_or_in_process_executor. You will need to use a different io manager to continue using this output. For example, you can use the mem_io_manager with the in_process_executor.
For more information on io managers, visit <https://docs.dagster.io/concepts/io-management/io-managers> 
For more information on executors, vist <https://docs.dagster.io/deployment/executors#overview>

Tim Castillo

04/17/2023, 3:16 PM

Hi! • Yeah, you'll have to use the

bigquery_resource

to execute a raw query against BQ • Looks like it's because it gave back a

RowIterator

object. I'm assuming that's a Python generator fn and not actual data, so you might want to either turn it into a dataframe with to_dataframe.

🙏 1

Akira Chang

04/17/2023, 3:21 PM

Hello! Thank you so much it worked!! 🙏🏼🔥

Akira Chang

04/17/2023, 3:26 PM

I defined the

Definitions

in my _`__init__.py`_ file. For example if I want multiple pipelines to use this same BigQuery resource, how would I be able to use it globally among different pipelines, in different folders?

Tim Castillo

04/17/2023, 3:27 PM

Hmm, are the pipelines in the same

Definitions

object?

Akira Chang

04/17/2023, 3:28 PM

Yes I think so

Akira Chang

04/17/2023, 3:29 PM

if they are not in the same Definitions object, then it doesn't work I assume?

Tim Castillo

04/17/2023, 3:29 PM

Yessirrrr

Akira Chang

04/17/2023, 3:33 PM

One more question, so my pipelines are in different folders, for example like:

Copy code

-My_project
 -Pipeline1 Folder
 -Pipeline2 Folder
 -Pipeline3 Folder
 -__init__.py

Akira Chang

04/17/2023, 3:33 PM

Do I put the Definition object in the

init

file like this to use globally?

Akira Chang

04/17/2023, 3:34 PM

Or do I need to put it somewhere under a folder?

Tim Castillo

04/17/2023, 3:38 PM

Yeah, put the

Definitions

object in the

__init__

.py and load all your pipelines into that

Definitions

object. For example, if they're assets, you can import the whole module and use

load_assets_from_module

Akira Chang

04/17/2023, 3:39 PM

Okk! Got it thank you!

4 Views

Open in Slack

Previous Next