What are the options to have an asset with historical data, which should only get refreshed manually, not to block any of its downstream assets from auto-materializing? These downstream assets should auto-materialize when other parent assets are updated, but never wait for assets which have historical data and which will almost never run.

To achieve the behavior where a downstream asset auto-materializes when other parent assets are updated, but does not wait for a specific parent asset with historical data that is only refreshed manually, you can use the auto-materialization policies provided by Dagster. Specifically, you can configure the auto-materialization policy of the downstream assets to skip materialization based on certain conditions related to their parent assets. Here are some options based on the documentation: 1. Selective Auto-Materialization Based on Parent Updates: You can use the

AutoMaterializePolicy.eager()

policy to auto-materialize a downstream asset whenever its parent assets are materialized. However, if you want to exclude a specific parent asset from triggering this auto-materialization, you might need to use a custom rule or filter. The documentation mentions that there is currently no explicit way to ignore updates from particular parents for auto-materialization, but this feature is on the radar. 2. Skip Materialization Based on Parent Materialization: You can use rules such as

AutoMaterializeRule.skip_on_not_all_parents_updated()

to skip materializing a downstream asset if not all of its parents have been updated. This rule can be configured to require all upstream partitions in each upstream asset to be materialized since the downstream asset’s last materialization in order to update it. 3. Manual Materialization: For the asset with historical data that should only be refreshed manually, you can avoid setting an auto-materialization policy. This way, it will not automatically trigger downstream materializations. Downstream assets can still have their own auto-materialization policies that respond to updates from other parent assets. 4. Custom Logic in Asset Definitions: While not explicitly mentioned in the documentation, you could potentially implement custom logic within your asset definitions to check the state of upstream assets and decide whether to proceed with materialization. It's important to note that some of these features are marked as experimental and the APIs may change in the future. Additionally, if the current auto-materialization policies do not fully cover your use case, you may want to follow the discussions and feature requests on the Dagster GitHub repository, as enhancements to these policies are considered based on user feedback and engagement. For the most accurate and up-to-date information on how to configure auto-materialization policies and rules, please refer to the official Dagster documentation and consider reaching out to the community or maintainers for specific feature requests or guidance. Skip Materialization When Parent Data Ve... button Is there a way to ignore updates for par... button Allow for fine grained rules/configurati... button Auto-Materialize if the direct upstream ... button Thank you for your feedback!

Regarding option 3, if this is for a dbt asset, which has the eager auto-materialization policy applied for all models in a folder, is there a way to specify in the config for that one model that it should not have any auto-materialization policy? @Scout: could you show me how to set it up in dbt?

In the context of dbt assets within Dagster, the auto-materialization policy can be configured using the

meta

configuration within your dbt project. This allows you to specify auto-materialization policies at the model level within your dbt project file (

dbt_project.yml

). To specify that a particular dbt model should not have any auto-materialization policy, you would need to set the

auto_materialize_policy

to

None

or simply omit it for that specific model. However, based on the documentation, there is no direct mention of setting the auto-materialization policy to

None

within the dbt configuration. Here's how you might typically configure an auto-materialization policy for dbt models using the

meta

configuration:

yaml models:   my_project:     my_model:       +meta:         dagster:           auto_materialize_policy:             type: eager

For the model that you do not want to auto-materialize, you would omit the

auto_materialize_policy

configuration:

yaml models:   my_project:     my_model_with_no_auto_materialization:       # No auto_materialize_policy specified here

However, if all models in a folder are set to have an eager auto-materialization policy and you want to exclude one model from this policy, you might need to explicitly set the policy for each model and omit it for the one you want to exclude. Please note that the ability to set

None

or a similar value to explicitly disable auto-materialization for a specific dbt model is not clearly documented. If this functionality is not currently supported, it may be a feature request that you could raise with the Dagster team. For the most accurate and current information on configuring auto-materialization policies for dbt assets in Dagster, please refer to the official Dagster documentation or reach out to the Dagster community for assistance.