The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Q: Are there any ongoing work to support Apache Arrow Flight (<https://arrow.apache.org/docs/format/Flight.html>)?

Realtime streaming of data in the pipeline while the flow is running, to/from Kafka, say. It can be to/from any stages in a pipeline

Dagster isn't really a streaming data framework, it's a DAG runner.

Streaming tools also involve a DAG but it's not a series of steps, it's more of a connection diagram for services that are always running.

I think another type of support could be data on the background so one does not need to pass data between solids

the I/O manager could be an interesting way to support apache arrow flight

Yeah, something treating it more like a database would be close to on-track for Dagster I think :slightly_smiling_face: