Hi team, is there any benefits using software-defined asset over using ops with AssetMaterialization?
05/06/2022, 3:38 PM
Hi Son! Great question.
Using ops with AssetMaterializations provides a lot of value, in terms of recording the history of what has happened to a particular asset, powering the historical views available in dagit. However, since these AssetMaterializations are all emitted at run-time, they are mostly useful for viewing what has happened in the past.
Using software-defined assets, by mapping the dependencies between assets, enables a richer set of interactions. You power the same historical views, but you now enable a set of operational tools, like being able to rematerialize an asset without mapping it to an op. It also enables you to attach rich metadata (like schema) to the asset, for a richer experience in Dagit. I think we’re just scratching the surface of what types of experiences we can build using software-defined assets.
Some other concrete advantages with software-defined assets:
• Before executing an op or job, you can know what assets it’s going to produce.
• You can navigate to the page for an asset and click a button to materialize it (and its ancestors)
• You can view a dependency graph of all assets automatically (without having to explicitly build it), since each asset knows about the assets it depends on.
05/07/2022, 3:29 PM
Thank you, could you send me some links/resources on how to explicitly build assets dependencies graph using AssetMaterialization as you mentioned?