Emmanuel Ogunwede01/12/2023, 1:44 AM
sandy01/12/2023, 9:54 PM
1. Are all the steps in my pipeline good candidates for software defined assets in dagsterAny step that produces an output in persistent storage is a good candidate for a software-defined asset.
1. Is it an anti pattern to pass the monthly partition definition to all of my assets (even if some assets can do without the partition definition) and just have one job and one scheduleAre the objects that these downstream assets are creating partitioned by month? I.e. would you like the ability to overwrite just the data for a particular month without affecting the data for other months? If yes, then it's a good fit for partitions. If no, then something you could do is have a non-partitioned asset downstream of the partitioned asset. Partitioned and non-partitioned assets can go inside the same scheduled job.
Emmanuel Ogunwede01/13/2023, 6:31 PM