Can you run real-time feeds in dagster? say I have a data pipeline that has 100 messages a second that need to be parsed and saved to the database.. Thanks!

hi @Vladyslav Ivanov! I think this might have been addressed in an above thread, but the basic answer is that in Dagster you would model this as a sensor which looks for new changes every N seconds, kicking off a job to handle those new changes.

Interesting. So you are doing batch processing... Not real-time I assume

yep exactly, there's no native support for streaming workloads in dagster, so you'd have to model it as "micro batches"

I see! Thank you so much, Owen