https://dagster.io/ logo
#ask-community
Title
# ask-community
v

Vladyslav Ivanov

06/02/2022, 5:09 PM
Can you run real-time feeds in dagster? say I have a data pipeline that has 100 messages a second that need to be parsed and saved to the database.. Thanks!
o

owen

06/02/2022, 7:59 PM
hi @Vladyslav Ivanov! I think this might have been addressed in an above thread, but the basic answer is that in Dagster you would model this as a sensor which looks for new changes every N seconds, kicking off a job to handle those new changes.
v

Vladyslav Ivanov

06/02/2022, 7:59 PM
Interesting. So you are doing batch processing... Not real-time I assume
Thank you for your help, Owen!
o

owen

06/02/2022, 8:00 PM
yep exactly, there's no native support for streaming workloads in dagster, so you'd have to model it as "micro batches"
v

Vladyslav Ivanov

06/02/2022, 8:01 PM
I see! Thank you so much, Owen