Hi. We have a message queue. Each message represen...
# ask-community
y
Hi. We have a message queue. Each message represents a resource event (crud action). We have a job with assets representing all of the resources. We want to create a sensor that will check the latest events and run a job with ops (assets ideally) selection only for the updated resources. Is there any way to do it using the current sensors/jobs API? Or maybe there is another way doing near-real-time assets updates?
j
This sounds feasible- the sensor would consume from the queue and emit run requests with different config per the contents of the messages
Depending on the api for the message queue, you might want to use the Sensor cursor to represent message index etc., or you may not use it at all
y
Great, thanks! And what about jobs ops / assets selection. Can I select them dynamically from the sensor?
Because I cannot find any suitable example in the docs
j
Ah, it looks like that’s currently not possible. There is a similar option to select a certain job: https://docs.dagster.io/_apidocs/schedules-sensors#dagster.RunRequest.job_name
@Dagster Bot issue Set op selection in RunRequest
d
y
So for now the only option is to have a list of pre-defined jobs for various ops / assets combinations, right?
j
That’s correct
y
And what about partitioned jobs? Can I submit a partitioned job run from the sensor?
115 Views