Stephen Bailey01/20/2023, 2:17 PM
events. One of the things we've encountered is that it's challenging to test these sensors (known problem), but also to trigger all of the downstream events in the case where we need to manually kick things off. We've created a
job that takes in an arbitrary asset key and generates an event for it, and it's been helpful for testing and other purposes. This might be a nice thing for Dagster to provide out of the box (attached to sensors, maybe?).
chris01/21/2023, 12:24 AM
Stephen Bailey01/21/2023, 4:08 PM
is materialized, then 5 or more sensors get triggered. Having to go to each of those sensors and trigger them defeats the point of the sensor in the first place -- the best remediation is to rematerialize the asset and let downstream processes pick up. I'm thinking more of a flow where you "Mark as Materialized" and you could manually (or programmatically) generate a materialization event. Use cases include: • marking an asset as materialized when a failure happens, and a manual fix has been taken. • marking when the bulk of computation has been done, and something non-essential fails in the op (I have hit this with adding
, for example) • allowing an external system to mark an asset as materialized (import / push events) in any case, i do think the focus is not on the sensor, but on the ability to manage the dagster event stream more directly
Dagster Jarred01/28/2023, 12:53 AM