Hi, out of curiosity, are there anyone here have u...
# announcements
x
Hi, out of curiosity, are there anyone here have use cases where you need to connect streaming and batch jobs? (Trigger batch jobs based on streaming jobs and monitor them together) And does dagster provides a way to do it?
m
Hi, Xinbin! You can trigger jobs using the GraphQL API currently, but this isn't packaged up nicely yet. Triggers are one of the things we're probably going to work on over the rest of this year, and we'd love any color you can provide on your needs.
s
hi, sorry for hijacking the conversation. I have a use-case for triggering pipelines from outside. I'm working for a survey company and we store responder data on kafka, when the survey filled there is an event on kafka, which can be the trigger for the Dagster pipeline to generate reports and process the collected data. Actually we are pretty close to implement this, so I'm very interested in what's the proper way to do this. Our current idea is to spin up a small service which process Kafka events and call out to Dagster whenever there is a new event.
m
that makes a lot of sense, there are also probably approaches that would use lambda etc
but we would love to know your trigger wish list