https://dagster.io/ logo
Join Slack
Powered by
Hello! I am currently making a pipeline where I ne...
# integration-bigquery
a
Hello! I am currently making a pipeline where I need do send message to Slack whenever a table in my 
Bigquery
is updated(a row is added). I'm wondering if I could use 
sensor
to monitor new changes to my table, and perhaps use 
bigquery_resource
in an 
asset
to retrieve data from Bigquery?
j
a
Ok! Thank you so much! I'm not quite sure how sensor works, so basically I define my 
asset
with 
bigquery_resource
that retrieves a table in dataframe format, and put that 
asset
inside a 
job
. And then put that 
job
in a 
sensor
and it will automatically run whenever theres a change in my table?
j
not quite. What you’ll write your asset that updates the bigquery table. Then you would specifically write an asset_sensor that will only run when the corresponding asset is materialized. In the body of the sensor you can connect to bigquery and confirm that the table was updated, and then send the slack message https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#defining-an-asset-sensor
a
Hello! So 
asset_sensor
will only be triggered when an 
asset
is materialized?
I'm thinking if my use case is sort of similar to this thread? Basically I have an asset that retrieves Bigquery data as a dataframe(I don't update anything on to BigQuery, I simply observe if any rows are added to the Bigquery from other sources) and use 
Sensor
and 
AssetObservation
to monitor if there are any update to the table periodically? https://dagster.slack.com/archives/C01U954MEER/p1679503786782659?thread_ts=1679476261.785759&cid=C01U954MEER
j
ok i think i understand better now. so scratch the asset sensor, but could you instead only have a plain sensor and within that sensor you create the bigquery connection, query the db, and send the slack message? there wouldn’t be any assets, ops, or jobs in this case
a
yepp!
So I dont need an asset for Bigquery
I directly have the query api in Sensor
j
yeah if you’re just checking the number of rows in a table that already exists, i don’t think you need an asset. you could do the query directly in the sensor
a
I see and the sensor has to have a "job" defined as well right
j
it’s optional!
a
Ill give it a try!
Thank you for the advise yesterday! It worked! 🙏👍
7 Views
Open in Slack
PreviousNext
Powered by