I have an asset_sensor that triggers a job to create a partitioned asset. I'd like to make the sensor configurable with a boolean, depending on which only the last partition is materialized or all non-materialized partitions are processed.
Currently, I cannot find a means to configure a sensor definition. What's the best way to provide a config to a sensor?
08/15/2022, 7:11 PM
Hi @Thorsten Schäfer - when would you ideally provide the config? Would it be at "definition-time" - i.e. inside your code? Or something you'd be able to tweak through Dagit?
08/15/2022, 7:14 PM
Hi @sandy, the latter is the case: ideally I'd have a boolean config property "backfill" in Dagit - if false, the sensor only starts a single run for the latest non-processed partition, otherwise it'd initiate a couple of runs, one for each non-processed partition.
08/15/2022, 8:24 PM
I see. Alas, there's no straightforward way to do this now. One thing you could do is have two sensors: one with each behavior. And you could switch one off and the other one on when you want it to switch
I filed an issue to track this request: https://github.com/dagster-io/dagster/issues/9378