is dagster suitable for stream processing Are there resource

Question

is dagster suitable for stream processing Are there resources for this

owen · Answer

hi < Abhinav Ayalur> quick answer is that Dagster is not designed for stream processing the closest approximation would be more of a micro batch architecture you could set up a sensor to listen for new events every N seconds then kick off a run that would process those X new events

Abhinav Ayalur · Answer

Would it be possible to store the state of the previous micro batch

Abhinav Ayalur · Answer

also would a way around this have my first op continuously yield outputs while listening to some endpoint

owen · Answer

for the first one sensors have a cursor object just a string but you can have a dictionary and json dump it if you have fancier use cases which can be updated on each execution

owen · Answer

for the second one not really op execution needs to complete before downstream steps start

owen · Answer

although to be honest the sensor constantly listening and kicking off runs is pretty similar in concept

Abhinav Ayalur · Answer

yea true

Abhinav Ayalur · Answer

as long as things are cached somewhere its possible

Abhinav Ayalur · Answer

can u have sensors run fast like sub second

owen · Answer

mmm i wouldn t really recommend running them much faster than once per 10 seconds or so default is 30 seconds