https://dagster.io/ logo
k

Klaus Stadler

02/05/2021, 8:51 AM
I wonder if I missed something in the docs... but I didn't find anything about how to coordinate the launch of multiple pipelines... e.g. I've Pipelines A and B and want to trigger Pipeline B right after Pipeline A is finished. How do you do this?
j

Jason

02/05/2021, 9:39 AM
I believe the idea is that Pipeline A should have some sort of external asset which is created or updated (state changes) and then you use
Sensors
to trigger Pipeline B. Doc links below but the blog post has a better example of Team A creating an asset which Team B relies upon prior to triggering a pipeline https://docs.dagster.io/overview/schedules-sensors/sensors
k

Klaus Stadler

02/05/2021, 9:44 AM
So there is no direct mechanism for pipelines that are in the same repository anyways? e.g. calling pipelines from a meta-pipeline?
j

Jason

02/05/2021, 9:56 AM
Not that I'm aware of
r

Ryan

02/05/2021, 10:03 AM
We're considering doing this by communicating with Dagster via GraphQL from inside a solid, perhaps the final solid in a pipeline, and "manually" kicking off another pipeline run. Can't attest to a successful experiment yet, but seems plausible 🙂
d

daniel

02/05/2021, 12:28 PM
Sensors are one recommended way to do this today as Jason says - if you want to compose two pipelines in the same repository into a single larger pipeline, this upcoming feature will allow you to do that directly: https://github.com/dagster-io/dagster/discussions/2902 .
👍🏾 2
k

Klaus Stadler

02/05/2021, 12:54 PM
Thats great 🙂 What is the currently expected release date for 0.11?
s

sandy

02/05/2021, 4:07 PM
@Klaus Stadler I believe we are aiming for some time in March. Out of curiosity, why separate these into two pipelines? Is it mainly for organizational purposes? Do you sometimes want to run one of them without the other one.
e

esztermarton

02/05/2021, 5:43 PM
We certainly have this use case that you mention @sandy (sometimes wanting to run one without the other, also, wanting it to halt at a particular point based on some kind of external or human driven condition - at other times permanently stop at a certain point)
👍 1
s

sandy

02/05/2021, 8:16 PM
Thanks @esztermarton - that makes sense
k

Klaus Stadler

02/08/2021, 8:51 AM
Hi @sandy, our case is pretty much the same
s

sephi

03/09/2021, 6:46 PM
likewise