https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alex Hunsberger

08/31/2023, 9:00 AM
Hi folks. Question regarding sensors - From https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors I can see an example sensor which polls and checks a directory for new files. I want to do something similar, but it's not safe to only check the existence of a file. I need to know that the file has finished being written to. In my case, I can know this by listening for a Linux inotify CLOSE_WRITE event (via the ipynotify library). The tricky part about integrating this use case with Dagster is that these notifications are event-based and not polling-based. I need a long running (Python) process or thread that is listening for inotify events, and the Dagster sensor would need to do something like .. check for any new events in a queue (I would create in memory), and start a job to process any newly written files. My specific question is .. what mechanism within Dagster architecture can I use to build something like this, having a long-running event-based process that writes to something that the Dagster sensor can read from? Thanks a lot for your help and please let me know if I should be posting in a different channel, etc.
d

daniel

08/31/2023, 3:20 PM
Hi Alex - right now our sensors only have support for polling. One option though would be to integrate more directly with some other event-based process and launch runs from within that process over our graphql API though: https://docs.dagster.io/concepts/webserver/graphql#launch-a-run
i.e. take on more of the responsibility of hosting the long-running python process, and only use dagster when it comes time to launch the run
a

Alex Hunsberger

09/01/2023, 1:18 AM
Thanks Daniel