run_key, sensors and idempotence. Hi, is there a n...
# announcements
run_key, sensors and idempotence. Hi, is there a notion of ‘successful execution’ that can be leveraged inside sensors? I am interested in the case where a sensor triggers the relevant pipeline but because - say - some other artefact is not yet available (or some other condition is not met), then the run is marked as ‘void’. The sensor is then expected to trigger again later. How can this be done? Thanks
Hi Marco - the way I would think about implementing this: • Include a solid in your pipeline that either fails or doesn't execute at all for "void" runs • In your sensor, use methods on
to find whether the that step failed or ran in the relevant pipeline runs. We could do a better job documenting and exposing the methods on the instance, but here they are: You may need to use semi-private APIs. I believe you can do something like
Copy code
step_stats = context.instance.get_run_step_stats(run_id, step_key)
status = step_stats.status

if status.value == "SKIPPED": # or "FAILED"
    # do something
thanks Sandy, that’s helpful. As a follow up question: how does the status ‘SKIPPED’ or ‘FAILED’ come about? In other words how do I ‘fail’ the solid?
You can do:
Copy code
from dagster import Failure

def my_solid(_):
   yield Failure()
Got it, thanks!
As I was trying to implement this I realised that either I don’t understand your approach or there is an issue. Regardless of status in the last run the run_key used in RunRequest will be already in the db, no?
So it’s won’t run again regardless?
Ah - that's right. You'd have to set
or use a unique run key if you wanted to be able to do two different submissions
Thanks. What’s the best way to debug a sensor, by the way?
How can I call a sensor so I can step in?