Henri Nguemaleu
03/06/2024, 1:30 PMScout
03/06/2024, 1:30 PMpython @asset def my_incremental_asset(context, config): if config["incremental"]: # Perform incremental update ... else: # Perform full refresh ...
In this example, the config
parameter can be used to determine whether the asset should be updated incrementally or fully refreshed. The actual logic for performing the incremental update or full refresh would be implemented within the asset function. Additionally, the documentation mentions that it could make sense to support incremental updates in a more first-class way, such as running a "backfill" over one of these assets which could trigger the full refresh. For more complex scenarios, such as assets that depend on themselves over time, the documentation suggests using a partition_mapping
or explicitly loading the previous value of the asset using load_asset_value
within the asset definition. Remember that the specific implementation details will depend on your use case and the nature of the data and computation involved in updating the asset.Henri Nguemaleu
03/06/2024, 1:32 PM