Hi Support, I'd like to use dagster as a tool to automate stuff between online services, e.g. TEAMS, GITHUB, ClickUp, ... I know this is not the original usecase of dagster, but we are users of dagster for ETL, so why using yet another tool. The only thing I see missing to get my use case implemented is the ability to listen to webhooks. I'd like to have some kind of webhook-sensor which NOT works via pulling but constantly listens to new requests. Ideally it would receive some json payload and forward it to the jobs triggered. Anything like that planned or some kind of workaround available? I am aware of the dagit GraphQL interface. But having dedicated webhook sensors would be much more easy/clear and manageable.

I had a fairly similar use case a few weeks ago, my solution was to send the payloads from the webhook to a service like SQS and use dagster sensors to poll it periodically. Not ideal, but works well enough for the PoC.

Thanks for the suggestion. It could be close to instant if the sensor polling would be low like 1sec. But over all this does not seem like an acceptable workaround for a production system. Hmm...

I don’t need super low latency in this specific pipeline, so it’s fine by me, why exactly do you not wanna use the GraphQL though?

Honestly I don't know how I would set this up e.g. in combination with ClickUp. I can give Clickup an URL it calls when a certain event happens, It then sends a json payload. I don't see how I could build a proper GraphQL query. Also I don't want the dagit GraphQL to be accessible to the whole inernet.

There’s definitely other ways to do this, an item I have in my backlog (so can’t say it works yet) is to test exposing a lambda with a webhook that will transform the request received into a GraphQL query. If you have your lambda inside the same VPC and expose dagit inside a private hosted zone/Cloud Map, you should be able to reach it from the lambda. Following this, I’d wind up having two dagit services: one behind cognito that’s accessible from the internet and a second one without authentication that’s only available from the VPC.

@Vinnie thanks for your help. I have a look into that but honestly it all reads like sub optimal workarounds. I guess I'll vote for this and keep hoping: https://github.com/dagster-io/dagster/issues/8096