The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

We're soon going to start using rabbit MQ for some event driven data processing. I'm not familiar with the area at all. Is Dagster well set-up for managing the ingestion side of that? Or would I have to use some other tool?

You could set up Dagster to trigger workflows on some event or combination of events, but you'd really need to think through how much volume you'd be expecting. We have Confluent and use Dagster to kick off some messaging services (Do X if some query Y results in records), and I'd be comfortable having it kick off short-running jobs too. Since Dagit has some default scheduling limits (&lt;100 concurrent tasks), you want to make sure not to swamp all your scheduled tasks with event-driven ones.

But if you're looking to, e.g. load the data using Dagster, I would recommend landing it in S3 first and loading it hourly, or setting up something like Snowpipe to handle the loading in real-time for you. We do the former with Segment and the latter with Confluent

I think sensors have a minimum interval time of 30s as well, so that is as fast as you're going to get

Interval time is configurable if you need faster ticks 