https://dagster.io/ logo
Title
o

Oliver

05/06/2021, 8:48 AM
hi is there any way to conditionally disable a sensor? for my usecase I have two pipelines each triggered by their own sensor. the first pipeline consumes from something like kafka, it checks if a record needs to be annotated, if not it sends to the database, if so it sends to a rabbit mq queue. the second pipelines sensor reads from the queue and sends records to an annotation service. it includes retry functionality so that if there is a problem with the annotation service it will exponential backoff and keep retrying until eventually it gives up. I would like to disable this sensor if there are too many jobs waiting on backoffs. i.e if the annotation service goes down
a

alex

05/06/2021, 3:25 PM
hm im not sure we have an api for that. Is it possible to include this logic in the sensor?
o

Oliver

05/07/2021, 5:19 AM
hmm, yea I can probably run a health check every n iterations or something and return if it comes back bad