Jonathan Kupcho

07/02/2021, 1:27 PM
Hello, I’m just starting to look at dagster so excuse the ignorance, but if you wanted to run pipelines through an event driven approach, would the idea be to implement a sensor? In this case, the event bus would be kafka. I’ve tried searching around for an approach and I’m coming up empty, I could also be trying to square peg a round hole.
:dagster: 1


07/02/2021, 2:26 PM
cc @prha


07/02/2021, 3:59 PM
I’m not a kafka expert, but the sensors are run at an interval in their own process. They’re designed to spin-up, evaluate to check state, and shut down quickly (to poll some external source). I think that’s not that compatible with having a long-lived kafka consumer subscribe to a kafka topic. We’ve discussed having some “trigger” API where you could easily post a message to the dagit service (over HTTP) and kick off a pipeline, but haven’t made a real push for it. Short of that, you could always trigger an execution using a GraphQL query to the Dagit server (similar to how the Dagit UI kicks off a pipeline execution)

Jonathan Kupcho

07/02/2021, 5:04 PM
Awesome, thanks for the response! That definitely clears it up for me.
:rainbow-daggy: 1
Kafka is intended to be long lived, you are correct. You can operate in a batch (with startup / shutdown), but not necessarily intended that way, and can have adverse effects on the broker, I believe.
That said, confluent themselves support a product that allows for REST driven consumers, so who knows!