Hi All —
Here’s my scenario:
• Two pipelines, Pipeline 1 appends data to a table (Table 1) daily and Pipeline 2 performs some aggregation on that table (Table 1) at the end of the month.
• Pipeline 2 should ALWAYS execute after Pipeline 1 runs on the last day of the month, and ONLY if Pipeline 1 runs successfully
Any suggestions on what to do regarding scheduling/sensors?
10/03/2022, 4:53 PM
Hi Isaac - are both or either of the pipelines partitioned? Also, is this using software-defined assets or op jobs?
10/04/2022, 3:43 AM
Isaac, you can materialize a asset at the end of your last op from the first Pipeline and then configure a sensor to run every 30 days and it'll search for the asset materialization that your first Pipeline did, when it finds the key, the run of the second Pipeline will starts
I also recommend you create a partition configuration for every last day of the month and create the asset condition in there so pipeline 2 only will execute at the end of the month, if the pipeline 1 executed just fine