https://dagster.io/ logo
Title
m

Mikołaj Gradowski

11/20/2021, 11:39 AM
Hi! I have the following problem — I have two graphs that fetch data from a few data sources for many graphs downstream. One job takes a few hours, but can start at 3AM. The second one is quick but can only start after 7AM. I would like to combine all graphs into a big daily super-graph, but i can't come up with an elegant solution of starting parts of this super-graph at different times. Any ideas? I considered just sleeping until 7AM in the second job, or requesting a restart after
7AM - time()
seconds. What bothers me with these is the unnecessary coupling of getting the data and scheduling.
m

max

11/21/2021, 11:58 PM
you could have different jobs built on top of sub graphs and scheduled separately
m

Mikołaj Gradowski

11/22/2021, 11:02 AM
I could, however the data dependency would not be modeled properly. I dug deeper and found Asset Sensors which are (almost) what I need. Thanks for your time!