David VanBolt11/12/2022, 4:54 PM
Stephen Bailey11/13/2022, 6:28 PM
to do this cleanly. It doesn't make things any less efficient from a coding perspective -- the main tradeoff is that it will generate a large number of nodes in your asset graph / clutter up your dagit ui / run history with hundred of different runs/assets. However, when you need the extra control (for example running a backfill on a single table), you have it naturally. Additionally, the dagster "repo" functionality allows you to exclude certain repos in the ui, which helps with the clutter problem. You could also do the "single job" approach, and yield AssetMaterialization events at the end of the run to populate your asset catalog with metadata. This is actually what we do currently (but wouldn't do again). You lose some of the nice catalog UI functionality, and don't really gain anything IMO from an end user perspectvie.
David VanBolt11/14/2022, 8:53 PM
Joris Ganne12/29/2022, 1:21 PM