Is there a recommended approach to stream all types of pipeline event types per run I.e Run Start / Step Start?
08/01/2022, 5:57 PM
hi @Matt O -- by "stream", do you mean stream them to some external system? You can query the event database for different event types, using
(inside either an op or a sensor).
08/01/2022, 6:02 PM
Hi @owen! Yes, I want to publish events to a queue as they happen
08/01/2022, 6:07 PM
Got it -- there's no direct way to stream these records, but you could definitely run that query on a pretty tight loop. One option would be to create a single-op job which runs that query and publishes all new records to your queue, and run that job every 30 seconds (or faster, if you have more real-time requirements).
08/01/2022, 6:13 PM
Got it, I'll look into this and test it out. Thanks for the insight 👍👍