is dagster suitable for stream processing? Are there resources for this?
12/06/2022, 1:02 AM
hi @Abhinav Ayalur! quick answer is that Dagster is not designed for stream processing, the closest approximation would be more of a micro-batch architecture (you could set up a sensor to listen for new events every N seconds, then kick off a run that would process those X new events)
12/06/2022, 1:05 AM
Would it be possible to store the state of the previous micro-batch?
also would a way around this have my first op continuously yield outputs while listening to some endpoint?
12/06/2022, 1:15 AM
for the first one, sensors have a cursor object (just a string but you can have a dictionary and json dump it if you have fancier use cases) which can be updated on each execution
for the second one, not really, op execution needs to complete before downstream steps start
although to be honest the sensor constantly listening and kicking off runs is pretty similar in concept
12/06/2022, 1:19 AM
as long as things are cached somewhere its possible
can u have sensors run fast? like sub-second?
12/06/2022, 1:38 AM
mmm i wouldn’t really recommend running them much faster than once per 10 seconds or so (default is 30 seconds)