https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Sehou Kouiho

04/06/2023, 1:50 PM
Hi all, I am new to Dagster and I am having some issue for a feature. In fact I would like to for new rows in a pandas table and launch a downstream job with the list of all new rows added to the table. I read that this could be achieve using sensor but I do not really know how to implement this. Anyone with an idea or help? To be more clear, can I asset the output of an asset from inside a sensor?
s

sandy

04/07/2023, 3:59 PM
Hi @Sehou Kouiho - there's no way to transfer potentially-large amounts of data between a sensor and a job. The easiest thing to do here would be to just have a job or asset that runs on some frequent schedule and reads new rows from the table. If you're worried about running this job too often, you could have a sensor look for new rows and only trigger the job if there are new rows. You could use a sensor cursor to track your current position in the table.
s

Sehou Kouiho

04/12/2023, 2:24 PM
Hi @sandy thank you for the tips. I implement a sensor that triggers a job on the new rows and it works perfectly. However, I still have one question. When I process the new rows, I would like the results to be appended to the ones of the already processed/seen rows. Actually, I try to implement a different I/O manager for the job running some processing on the new rows to insert into the existing result. I find this a bit cumbersome and I am wondering if you have any more efficient tips to implement such use case. Thank you.
2 Views