How should I solve this flow? I have an op (`op A...
# ask-community
How should I solve this flow? I have an op (
op A
) which does an api call that creates an item. (response is urn) Then I have a sensor which receives events on the progress of calculating the item (which happens outside our control) This sensor can receive events of multiple items. Then I want to continue the job which used
op A
because there is specific data in the job later on. I want to create something like a second op which can wait/ do something on an event by the sensor when the urn matches. I don't want to use polling as this would cause extra delays or resources when you make the polling delay low.
Hi Irven. Sensors in Dagster operate outside of the main job execution flow. In order to use sensors as you describe, you’d set up two separate jobs, one that does the API call and one that consumes events from the API (kicked off by the sensor). Alternatively, you could set everything up in a single job, but you’d have to set up an op that spins in a while loop while reading from the API. I’m not sure what you mean by
don't want to use polling
. In both scenarios, you would use polling to check status. If there’s a way to provide a callback to the API (push model), you could use the GraphQL API to kick off a job.
I will think a bit on this. (I have vacation next week, so will come back a while later 🙂 )
To give more background, we are using a wrapper around pulsar: So we have a general consumer listening on inputs. I would like to have some dispatcher to give it to the right op and that the op waits on events for a certain time. but probably I need to find another pattern