Hi team, thanks so much for the multi-job sensor feature 🙏  Looks like the sensor can either yield one SkipReason / multiple Run Requests. Since my sensor is targeting multiple jobs, I might want to yield multiple SkipReasons too. Is it something which will be changed in the future or there is a better way to do it today?

Hey Arun! Do you have something on your side that is consuming the structured

SkipReason

out of the tick history? Right now, the main purpose of

SkipReason

is to display some text on the sensor tick timeline in Dagit. Would you want a different behavior in how we display those messages if we did enable multiple

SkipReasons

?

Hi @prha, Nope, I don't have any setup that consumes SkipReason. I am asking this to provide more context on the Dagit UI. Each of our jobs has a set of dependencies and the sensor checks the dependencies statuses for all the jobs and triggers a job if all the deps are satisfied. In a single sensor evaluation there can be some jobs that passes the dependencies checks (and get a RunRequest) and some that don't (get a SkipReason). I want to show the SkipReason information on the UI even when there are RunRequests

Yeah, I think there would be less confusion with a 1:1 mapping of job/sensor. I’ve been thinking about SkipReason and RunRequest as scoped to a single sensor evaluation function. For each sensor evaluation, it can either kick off 1 or more runs, or it can provide a message that no runs were kicked off. As you say, run requests and skip reasons on a single tick might be a little confusing with this mental model. What is the ideal tool for tracking the history of multi-job sensors? Would you want individual timelines / pages for each job? And if so, is the main advantage of multi-job sensors over multiple individual sensors just that it saves computation in terms of execution? If it’s a matter of adding some structure in the messages, would it be sufficient to add some structured metadata to the skip reason?

Thanks for the explanation. Yes, the idea to have multi job sensors is to save some computation. So the sensor here does an API call to fetch the source metadata and see if the dependencies for the sources are satisfied. I think a single sensor fetching all the sources in a single API and checking their deps would be better than creating too many sensors considering the sequential execution of sensors. Another reason (not very important) is that the sensors are disabled by default now. In our case, if we do a 1:1 mapping, the sensors will be dynamically created (as the jobs are also created dynamically) and need a graphQL intervention to start the sensor. cc: @sandy (already had a similar conversation with him and settled on multi job sensor)

I think it depends on whether or not you need to support mixed SkipReason + RunRequest ticks. I think that’s a scenario that is not easy for us to handle at the moment.