https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Abhishek Tripathi

02/24/2024, 9:46 AM
how to auto materialise assets in a pipeline?
s

Scout

02/24/2024, 9:46 AM
To auto-materialize assets in a pipeline, you can use Auto-materialize policies (AMPs) in Dagster. AMPs allow you to assign policies to assets that let Dagster determine the best approach to keeping assets up-to-date while adhering to those policies. Here are some of the ways you can define AMPs to auto-materialize assets: 1. Auto-materialize policies replace the asset reconciliation sensor: We've made substantial improvements to the APIs used for specifying which assets are scheduled declaratively. AutoMaterializePolicies work across code locations and allow you to customize the conditions under which each asset is auto-materialized. 2. Auto-materialize policies and data versions: It's possible to auto-materialize stale assets that are downstream of an observable source asset. They use the source asset observations to determine whether upstream data has changed and assets need to be materialized. 3. Materialize this asset when the assets it depends on are all updated: You can set rules to materialize an asset when all its dependencies have been updated. 4. Materialize this asset at a specific time: For example, you can set a policy to materialize an asset every day at 8 AM. 5. Materialize this asset if it hasn't been materialized after a certain period: For instance, you can auto-materialize an asset if it hasn't been materialized in over an hour. These rules can be combined or negated to meet the expectations for when each asset needs to be updated, ensuring that data assets are updated precisely as needed rather than on a fixed schedule. Please refer to the [Auto-materialize policies documentation](https://docs.dagster.io/concepts/assets/asset-auto-execution) for more detailed information on how to implement these policies in your Dagster pipelines.
5 Views