HI team, I am looking to build inter-job dependenc...
# ask-community
h
HI team, I am looking to build inter-job dependencies and trigger downstream jobs when an upstream job succeeds. It seems that `run_status_sensor`() is the way to go. I see that it support monitored_jobs and monitored_pipelines. For monitored_jobs, could I reference a job by its string name instead of the job object?
🤖 1
s
This isn't currently possible, but we might be able to make it possible soon. Out of curiosity, what's the reason you'd prefer to use a string than a job object? Here's a related request that we got recently: https://github.com/dagster-io/dagster/issues/9394
e
there are a few other ways to do this, but nothing exactly like what you're saying. For example, it's possible to trigger a job using a sensor if you create a new object in a database. Dagster also has an “asset sensor” which does something similar. Main limitation is that it does not trigger a job for each new asset materialization, just the latest one (although there is an open MR to fix that here: https://github.com/dagster-io/dagster/issues/5699)
h
Thanks @sandy. I see the
run_status_sensor()
monitored_jobs param could take in an
UnresolvedAssetJobDefinition
. I am wondering if I could use
UnresolvedAssetJobDefinition
with an asset job name (or something?) to reference another job? (Context: we dynamically generate jobs based on yamls defined by our users. So when we are generating each job, we don’t have the complete references to all jobs.)
Hi @Eegan K, thanks for the pointer. If i set the limit to None and get back all records after prev cursor, is there a way for us to figure out which partition the asset materialization event is? Do I have to query with run_id for each event?
e
To be completely honest I’m not sure. In my use case the cursor wasn’t useful, so I didn't look into it.
h
I see. Thanks
s
@Hebo Yang it's possible that your
UnresolvedAssetJobDefinition
scheme would work, but you'd need to try it out to be sure. when this change is merged, we should have a more supported way of doing this: https://github.com/dagster-io/dagster/pull/9405
h
Got it. Thanks Sandy!