Hi all, I am new to Dagster and I am having some issue for a feature. In fact I would like to for new rows in a pandas table and launch a downstream job with the list of all new rows added to the table. I read that this could be achieve using sensor but I do not really know how to implement this. Anyone with an idea or help? To be more clear, can I asset the output of an asset from inside a sensor?

Hi @Sehou Kouiho - there's no way to transfer potentially-large amounts of data between a sensor and a job. The easiest thing to do here would be to just have a job or asset that runs on some frequent schedule and reads new rows from the table. If you're worried about running this job too often, you could have a sensor look for new rows and only trigger the job if there are new rows. You could use a sensor cursor to track your current position in the table.

Hi @sandy thank you for the tips. I implement a sensor that triggers a job on the new rows and it works perfectly. However, I still have one question. When I process the new rows, I would like the results to be appended to the ones of the already processed/seen rows. Actually, I try to implement a different I/O manager for the job running some processing on the new rows to insert into the existing result. I find this a bit cumbersome and I am wondering if you have any more efficient tips to implement such use case. Thank you.