https://dagster.io/ logo
#ask-ai
Title
# ask-ai
j

Joshua Greenhalgh

02/25/2024, 6:14 PM
When a collection of assets is turned into an AssetJob - how can we cache downstream assets in the job?
s

Scout

02/25/2024, 6:14 PM
To cache downstream assets when using an AssetJob in Dagster, you would typically rely on the built-in behavior of Dagster's execution engine, which is designed to avoid recomputing assets that have not changed since their last computation. However, the specifics of caching behavior can depend on the configuration and setup of your Dagster instance and the storage system you are using. The documentation provides guidance on defining AssetJobs and materializing assets, but it does not explicitly detail the internal caching mechanisms for downstream assets within an AssetJob. It does mention that in production, caching the dbt Cloud job compilation is recommended to reduce latency, especially as the size of the dbt project grows. If you are looking to ensure that downstream assets are not recomputed unnecessarily, you would typically ensure that the assets are defined with the appropriate dependencies and that the storage configuration is set up to track the metadata of asset materializations. Dagster's asset-aware execution engine will then determine whether an asset needs to be recomputed based on changes to its inputs or code. For more advanced caching strategies, such as custom caching logic or in-process execution caching, you might need to look into custom solutions or follow the ongoing discussions and feature requests in the Dagster community, as these may not be covered in the current documentation. If you have a specific use case or configuration in mind that is not addressed by the documentation, it may be helpful to provide more details or check the latest updates from the Dagster community for any new features or patterns that have been introduced since the knowledge cutoff date.