https://dagster.io/ logo
#announcements
Title
# announcements
a

Alex Despotakis

03/17/2021, 3:16 PM
Hey all, not to pile on the questions, but can anyone also tell me if there's a way to truly delete a sensor from dagster? I think I broke this one sensor that absolutely will not trigger because of some dagster configurations and want dagster to recognize it as a brand new sensor that has never ran before. Any help is greatly appreciated
a

alex

03/17/2021, 3:30 PM
I think I broke this one sensor that absolutely will not trigger because of some dagster configurations
Can you share some more details about you mean here, maybe a screen shot? @daniel / @prha
p

prha

03/17/2021, 3:50 PM
Hi @Alex Despotakis… are you using the run key to enforce idempotence? I think we query runs in the run storage for tags with the sensor name and the run_key pairing.
If you want to reset the state of the run key checks, I think you’d either have to rename the sensor, delete the runs for that sensor name (obviously destructive), or manually delete the tags for those runs in the
run_tags
table.
If your question is more about editing the tick history for a given sensor, then there’s not an easy way to do that on a per-sensor basis, though that seems very reasonable to do. If you file a github issue, we can see if we can add a CLI tool or an action in dagit that would wipe that history.
The manual way to clear this would be to delete the corresponding rows in the
job_ticks
table.
a

Alex Despotakis

03/17/2021, 4:02 PM
I can definitely explain it in excruciating detail, assuming I renamed my sensor. 1. I make a new sensor that looks at my filesystem for documents with a minimum interval of 30 seconds. The sensor finds the files in a folder called "to_process", yields the run requests, and triggers the pipeline. Once my first solid executes, it takes all of the files in "to_process" and moves them to a "processed" folder. The sensor continuously runs, yielding a skip since there are no more documents to process. The last tick is at 11:47am. I turn the sensor off. 2. I notice that for some reason, the files were not processed correctly. I re-populate the "to_process" folder with the same files and turn the sensor back on. As of 11:52am, the sensor has not been triggered. The sensor-daemon continuously returns these logs:
Copy code
2021-03-17 11:52:19 - SensorDaemon - INFO - Checking for new runs for the following sensors: PullDocsFromFileSystem4
2021-03-17 11:52:24 - SensorDaemon - INFO - Checking for new runs for the following sensors: PullDocsFromFileSystem4
2021-03-17 11:52:29 - SensorDaemon - INFO - Checking for new runs for the following sensors: PullDocsFromFileSystem4
2021-03-17 11:52:34 - SensorDaemon - INFO - Checking for new runs for the following sensors: PullDocsFromFileSystem4
2021-03-17 11:52:39 - SensorDaemon - INFO - Checking for new runs for the following sensors: PullDocsFromFileSystem4
2021-03-17 11:52:44 - SensorDaemon - INFO - Checking for new runs for the following sensors: PullDocsFromFileSystem4
3. I refresh the page, turn the sensor back off, then delete the latest runs in the dagster UI. I refresh the page again and turn the sensor back on. The sensor is still showing the latest tick of 11:47am with no indication of the next tick. Refreshing the metadata for the repository doesn't seem to do anything either. The sensor will infinitely stall and as of 11:57am, no new runs were triggered. The short term solution of how to trigger it is as @prha suggested: changing the name from
PullDocsFromFileSystem4
to
PullDocsFromFileSystem5
will work. The sensor will trigger like it was brand new; however, to have a continuously changing sensor name in a repo would not be ideal. Additionally, I am not using a run key. The sensor context object passed through the sensor isn't touched in any way. Also, thank you @prha for your responses! I will try deleting the corresponding rows in the
job_ticks
table and see if that fixes anything.
a

alex

03/17/2021, 4:11 PM
im a little confused by entry 3 - is there a “turned the sensor back on” that is missing? As it is currently written it sounds like the sensor is just in the off state.
a

Alex Despotakis

03/17/2021, 4:11 PM
Hey @alex, you're right. Making the edit now
Edit: 3. I refresh the page, turn the sensor back off, then delete the latest runs in the dagster UI. I refresh the page again and turn the sensor back on. The sensor is still showing the latest tick of 11:47am with no indication of the next tick. Refreshing the metadata for the repository doesn't seem to do anything either. The sensor will infinitely stall and as of 11:57am, no new runs were triggered.
a

alex

03/17/2021, 4:14 PM
and what version of dagster are you running?
a

Alex Despotakis

03/17/2021, 4:15 PM
I'm on version 0.10.7
👍 1
p

prha

03/17/2021, 4:24 PM
what do you get when you run
dagster sensor preview PullDocsFromFileSystem4
a

Alex Despotakis

03/17/2021, 4:26 PM
The sensor executes just fine. It yields the expected run requests, then exits.
2 Views