https://dagster.io/ logo
#ask-ai
Title
# ask-ai
h

Henri Nguemaleu

03/06/2024, 1:30 PM
is it possible to have a python asset incremental
s

Scout

03/06/2024, 1:30 PM
Yes, it is possible to have a Python asset that is updated incrementally in Dagster. Incremental assets are those that can incorporate recent upstream data into the asset, rather than recomputing the entire asset from scratch each time it is updated. The documentation provides an example of how to define an asset that can be updated incrementally or fully refreshed. This can be done by including a configuration option that controls whether to do an incremental vs. full-refresh update. Here's a snippet from the documentation that illustrates this concept:
Copy code
python @asset def my_incremental_asset(context, config):     if config["incremental"]:         # Perform incremental update         ...     else:         # Perform full refresh         ...
In this example, the
config
parameter can be used to determine whether the asset should be updated incrementally or fully refreshed. The actual logic for performing the incremental update or full refresh would be implemented within the asset function. Additionally, the documentation mentions that it could make sense to support incremental updates in a more first-class way, such as running a "backfill" over one of these assets which could trigger the full refresh. For more complex scenarios, such as assets that depend on themselves over time, the documentation suggests using a
partition_mapping
or explicitly loading the previous value of the asset using
load_asset_value
within the asset definition. Remember that the specific implementation details will depend on your use case and the nature of the data and computation involved in updating the asset.
h

Henri Nguemaleu

03/06/2024, 1:32 PM
can you give me the link of the article ?
4 Views