@Scout what is the minimal time interval that a Dagster sensor can detect a change ?

The minimal time interval that a Dagster sensor can detect a change is determined by the

minimum_interval_seconds

parameter that you can set when defining a sensor. This parameter specifies the minimum number of seconds that will elapse between sensor evaluations. For example, if you set

minimum_interval_seconds=30

, the sensor will not be evaluated more frequently than every 30 seconds. Here's an example from the documentation:

python @sensor(job=my_job, minimum_interval_seconds=30) def sensor_A():     yield RunRequest(run_key=None, run_config={})

In this example,

sensor_A

is configured to run at a minimum interval of 30 seconds. This means that the sensor will not be evaluated more frequently than every 30 seconds, but it could be evaluated less frequently depending on the sensor's execution time and the Dagster daemon's scheduling. It's important to note that this interval represents a minimum interval between runs of the sensor and not the exact frequency the sensor runs. If a sensor takes longer than the specified minimum interval to complete its execution, the actual time between runs will be longer. Additionally, if a sensor evaluation function takes more than 60 seconds to return its results, the sensor evaluation will time out, and the Dagster daemon will move on to the next sensor without submitting any runs. To avoid timeouts, slower sensors can break up their work into chunks, using cursors to let subsequent sensor calls pick up where the previous call left off.