@Scout how do I skip auto-materialization for backfills?

To skip auto-materialization for backfills, you can use the

skip_on_backfill_in_progress

rule provided by Dagster's auto-materialization policies. This rule is designed to skip the auto-materialization of asset partitions if they are targeted by an in-progress backfill. Here's how you can use this rule according to the documentation:

python from dagster import AutoMaterializePolicy, AutoMaterializeRule  my_policy = AutoMaterializePolicy.eager().with_rules([     AutoMaterializeRule.skip_on_backfill_in_progress() ])

This policy will ensure that asset partitions targeted by an active backfill will not be auto-materialized. You can further customize the behavior of this rule by setting the

all_partitions

parameter to

True

or

False

. If set to

True

, it skips all partitions of the asset being backfilled, regardless of whether the specific partition is targeted by a backfill. If set to

False

, it skips only partitions targeted by a backfill. Please note that this rule is part of the auto-materialization policy, and you would need to apply this policy to your assets for it to take effect.

@Scout can I do the reverse? Only auto-materialze for backfill runs?

The documentation does not explicitly mention a built-in feature or policy that would allow auto-materialization to occur only for backfill runs and skip all other runs. Auto-materialization policies in Dagster are generally designed to automate the materialization of assets based on certain conditions, but they do not provide a direct way to restrict materialization exclusively to backfill runs. If you want to achieve this behavior, you might need to implement custom logic within your asset definitions or orchestration code to check the context of the run and decide whether to proceed with materialization. This could involve checking for specific tags or run metadata that indicate a backfill run. However, since the documentation does not provide a clear solution for this specific use case, it's recommended to consult the latest Dagster documentation or reach out to the Dagster community for potential updates or workarounds that might have been introduced after the knowledge cutoff date.

@Scout can I write a custom AutoMaterializationRule?

As of the last update before my knowledge cutoff in 2023, the documentation does not provide a way to write custom

AutoMaterializeRule

classes directly. The

AutoMaterializeRule

API provides a set of predefined rules that can be used to control the auto-materialization behavior of assets. These rules can be combined and configured to create an

AutoMaterializePolicy

that suits your needs. The available rules include: - `materialize_on_cron`: Materialize an asset partition based on a cron schedule. - `materialize_on_missing`: Materialize an asset partition if it has never been materialized before. - `materialize_on_parent_updated`: Materialize an asset partition if one of its parents has been updated more recently than it has. - `materialize_on_required_for_freshness`: Materialize an asset partition if it is required to satisfy a freshness policy. - `skip_on_backfill_in_progress`: Skip an asset’s partitions if targeted by an in-progress backfill. - `skip_on_not_all_parents_updated`: Skip materializing an asset or partition if not all of its parents have been updated. These rules can be added to or removed from an

AutoMaterializePolicy

using the

with_rules

and

without_rules

methods, respectively. If the predefined rules do not cover your specific use case, you might need to implement custom logic outside of the

AutoMaterializeRule

framework. This could involve creating custom sensors or orchestration logic that triggers materializations under the conditions you define. For the most up-to-date information and potential new features, including the ability to write custom auto-materialization rules, please refer to the latest Dagster documentation or inquire with the Dagster community.