https://dagster.io/ logo
#integration-bigquery
Title
# integration-bigquery
a

Akira Chang

04/25/2023, 3:16 PM
Hello! I am currently making a pipeline where I need do send message to Slack whenever a table in my
Bigquery
is updated(a row is added). I'm wondering if I could use
sensor
to monitor new changes to my table, and perhaps use
bigquery_resource
in an
asset
to retrieve data from Bigquery?
j

jamie

04/25/2023, 3:19 PM
a

Akira Chang

04/25/2023, 3:38 PM
Ok! Thank you so much! I'm not quite sure how sensor works, so basically I define my
asset
with
bigquery_resource
that retrieves a table in dataframe format, and put that
asset
inside a
job
. And then put that
job
in a
sensor
and it will automatically run whenever theres a change in my table?
j

jamie

04/25/2023, 5:51 PM
not quite. What you’ll write your asset that updates the bigquery table. Then you would specifically write an asset_sensor that will only run when the corresponding asset is materialized. In the body of the sensor you can connect to bigquery and confirm that the table was updated, and then send the slack message https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#defining-an-asset-sensor
a

Akira Chang

04/26/2023, 9:11 AM
Hello! So
asset_sensor
will only be triggered when an
asset
is materialized?
I'm thinking if my use case is sort of similar to this thread? Basically I have an asset that retrieves Bigquery data as a dataframe(I don't update anything on to BigQuery, I simply observe if any rows are added to the Bigquery from other sources) and use
Sensor
and
AssetObservation
to monitor if there are any update to the table periodically? https://dagster.slack.com/archives/C01U954MEER/p1679503786782659?thread_ts=1679476261.785759&cid=C01U954MEER
j

jamie

04/26/2023, 2:48 PM
ok i think i understand better now. so scratch the asset sensor, but could you instead only have a plain sensor and within that sensor you create the bigquery connection, query the db, and send the slack message? there wouldn’t be any assets, ops, or jobs in this case
a

Akira Chang

04/26/2023, 2:48 PM
yepp!
So I dont need an asset for Bigquery
I directly have the query api in Sensor
j

jamie

04/26/2023, 2:49 PM
yeah if you’re just checking the number of rows in a table that already exists, i don’t think you need an asset. you could do the query directly in the sensor
a

Akira Chang

04/26/2023, 2:50 PM
I see and the sensor has to have a "job" defined as well right
j

jamie

04/26/2023, 2:53 PM
it’s optional!
a

Akira Chang

04/26/2023, 2:53 PM
Ill give it a try!
Thank you for the advise yesterday! It worked! 🙏👍
7 Views