Q Are there any ongoing work to support Apache Arrow Flight

Question

Q Are there any ongoing work to support Apache Arrow Flight <https arrow apache org docs format Flight html>

Noah K · Answer

What kind of support do you mean

bklau-zap · Answer

Realtime streaming of data in the pipeline while the flow is running to from Kafka say It can be to from any stages in a pipeline

Noah K · Answer

Dagster isn t really a streaming data framework it s a DAG runner

Noah K · Answer

Streaming tools also involve a DAG but it s not a series of steps it s more of a connection diagram for services that are always running

Daniel Fernandez · Answer

I think another type of support could be data on the background so one does not need to pass data between solids

Daniel Fernandez · Answer

the I O manager could be an interesting way to support apache arrow flight

Noah K · Answer

Yeah something treating it more like a database would be close to on track for Dagster I think slightly smiling face