https://dagster.io/ logo
#dagster-support
Title
# dagster-support
m

Makoto

07/30/2021, 11:44 PM
I have two pipelines, A and B. If I want B to run upon successful run of A, would using sensors be the right way to do it? I’m thinking pipeline A would have a solid to write a file to S3 to indicate it was done, and then a sensor would pick that up and runs pipeline B. Please let me know if there’s a better way.
p

prha

07/31/2021, 12:34 AM
Hi @Makoto! Yes, that’s a good way to set up that inter-pipeline dependency. We have asset sensors to check for materializations of certain external assets, which might be a particularly good fit for this use case. We’re also exploring ways of surfacing these inter-pipeline dependencies through assets in Dagit, so stay tuned for more updates on this front.
Should note that using asset sensors, pipeline A would not have to write to S3 or some external storage in order to trigger pipeline B, it would just need to yield an
AssetMaterialization
event with an
AssetKey
that the pipeline B sensor is listening for (though it could certainly do both).
👍 1
m

Makoto

07/31/2021, 12:53 AM
Thanks @prha!
Oh that’s good to know!
That seems like a better approach. Will give it a try
@Andrew Brown ^
c

chrispc

07/31/2021, 5:30 PM
Hi @prha, I want to use your approach but in my case, I have several pipelines (A, B, C, ...) to execute a pipeline Z ( It would consume the materializations of pipelines A, B, C, so on ). The pipelines A, B, C,... will have different schedulers at times/days. The pipeline Z only should be trigger if all the dependencies were success. Do you have something in mind to develop this? you said the pipelines can listening AssetKey, could the pipelines listen for more than one AssetKey and wait to get all the AssetKey needed?
p

prha

08/02/2021, 3:35 PM
@chrispc You can certainly have a sensor listening for more than one asset key, but you would have to write the logic that queries for the event records manually instead of using the
@asset_sensor
decorator. You could then update the cursor if and only if all of the A, B, C pipelines have emitted their corresponding materializations. I believe there’s some example code in the documentation for this: https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#multi-asset-sensors
c

chrispc

08/02/2021, 3:38 PM
thank you so much @prha
a

Andrew Brown

08/11/2021, 4:50 PM
@prha I’d like to use the new
@run_status_sensor
decorator to kick off a pipeline B upon successful completion of a pipeline A. Is that possible? I don’t see how yielding a RunRequest would work in this case, since the
@run_status_sensor
decorator does not take a
pipeline_name
parameter to target for requested runs.
2 Views