Hello guys I m new to Dagster using 0 9 18 and running insid

Question

Hello guys I m new to Dagster using 0 9 18 and running inside a container and I was wondering if it was possible to chain several pipelines For example I want to start my pipeline 2 as soon as pipeline 1 is completed Thank you in advance and have a nice day

Noah K · Answer

You could make a solid that hits the GraphQL API

Noah K · Answer

Though a think a single pipeline with a composite solid would be the more Correct way to handle it

Nicolas Gaillard · Answer

I don t quite understand It would mean that my global pipeline contains a composite solid which would contain two solids corresponding to my two pipelines

sandy · Answer

< Nicolas Gaillard> that s exactly right Out of curiosity why separate them into two separate pipelines They re in different logical domains so it s helpful to organizing them that way They have different sets of dependences You want to be able to run them at different cadences

Nicolas Gaillard · Answer

The objective is to process data from the same business domain collection transformation and exposure In the case where the provisioning doesn t work it forces me to restart the entire pipeline In addition I expose the data to different services and at different frequencies Finally in order to avoid a huge pipeline with a multitude of solids and even when performing abstractions using composite solids I find it reasonable to separate the pipelines What do you think about it

sandy · Answer

Got it Will get back to you with a code example soon

sandy · Answer

gt In the case where the provisioning doesn t work it forces me to restart the entire pipeline Would you mind expanding on this a little bit The issue is that you experience failures while provisioning for individual solids within a pipeline

Nicolas Gaillard · Answer

Awesome can t wait to read you

Nicolas Gaillard · Answer

Let s imagine that I have a daily pipeline that collects CSV data through a solid performs transformations with Pandas in another stores the data in a database and a last one that generates a view using this data For some reason my fourth solid may fail occasionally other data not ready for example It seems to me that I can t easily restart via the UI only this solid and therefore I have to restart the whole pipeline

sandy · Answer

Do you know what intermediate storage you re using The default is in memory which means that any outputs produced by the earlier steps will disappear between runs

sandy · Answer

I wrote up a code snippet here <https github com dagster io dagster discussions 3436> Let me know if you have any questions It s a little bit involved we re interested in making this simpler in the future Note that it depends on some internal APIs so it might get broken in a future release

Noah K · Answer

Having a libraries thing like dagster graphql solids might be nice

Noah K · Answer

Was kind of surprised there wasn t already one slightly smiling face

sandy · Answer

Yeah this has been one of most upvoted issues for a while <https github com dagster io dagster issues 2674>

Noah K · Answer

Though counterpoint it s very easy to write one myself smile

Nicolas Gaillard · Answer

Thank you for your messages and yes I do use in memory intermediate storage Thanks again for the snippet it looks great I ll try to integrate it and will get back to you if I have any questions or to give you feedback

sandy · Answer

If you switch to a different intermediate storage like the filesystem one you ll be able to kick off pipelines from the middle