Hey, what would be the recommended approach to subscribe to an events stream and trigger pipelines execution based on the events received? I have, in the same repo as my dagster pipelines code, a connector to the event stream so that when an event is received I trigger a pipeline with execute_pipeline (this just works locally), my doubt is how to do this once deployed to k8s for example, should I be calling dagit with graphql to trigger pipeline execution? . I believe sensors cannot be used for this purpose, or can they?

Hi ruben - if its a situation where can poll for updates on a fixed interval from the event stream then sensors will work, but they're not well suited for a long-running subscription that needs to keep a single process running for a long time. if that's a requirement then launching the pipeline via GraphQL when a new event comes in is probably the best option.

thanks a lot 🙂