https://dagster.io/ logo
#ask-ai
Title
# ask-ai
m

Marcel Steinbach

03/06/2024, 4:53 PM
@Scout how do I skip auto-materialization for backfills?
s

Scout

03/06/2024, 4:53 PM
To skip auto-materialization for backfills, you can use the
skip_on_backfill_in_progress
rule provided by Dagster's auto-materialization policies. This rule is designed to skip the auto-materialization of asset partitions if they are targeted by an in-progress backfill. Here's how you can use this rule according to the documentation:
Copy code
python from dagster import AutoMaterializePolicy, AutoMaterializeRule  my_policy = AutoMaterializePolicy.eager().with_rules([     AutoMaterializeRule.skip_on_backfill_in_progress() ])
This policy will ensure that asset partitions targeted by an active backfill will not be auto-materialized. You can further customize the behavior of this rule by setting the
all_partitions
parameter to
True
or
False
. If set to
True
, it skips all partitions of the asset being backfilled, regardless of whether the specific partition is targeted by a backfill. If set to
False
, it skips only partitions targeted by a backfill. Please note that this rule is part of the auto-materialization policy, and you would need to apply this policy to your assets for it to take effect.
m

Marcel Steinbach

03/06/2024, 4:54 PM
@Scout can I do the reverse? Only auto-materialze for backfill runs?
s

Scout

03/06/2024, 4:54 PM
The documentation does not explicitly mention a built-in feature or policy that would allow auto-materialization to occur only for backfill runs and skip all other runs. Auto-materialization policies in Dagster are generally designed to automate the materialization of assets based on certain conditions, but they do not provide a direct way to restrict materialization exclusively to backfill runs. If you want to achieve this behavior, you might need to implement custom logic within your asset definitions or orchestration code to check the context of the run and decide whether to proceed with materialization. This could involve checking for specific tags or run metadata that indicate a backfill run. However, since the documentation does not provide a clear solution for this specific use case, it's recommended to consult the latest Dagster documentation or reach out to the Dagster community for potential updates or workarounds that might have been introduced after the knowledge cutoff date.
m

Marcel Steinbach

03/06/2024, 5:04 PM
@Scout can I write a custom AutoMaterializationRule?
s

Scout

03/06/2024, 5:04 PM
As of the last update before my knowledge cutoff in 2023, the documentation does not provide a way to write custom
AutoMaterializeRule
classes directly. The
AutoMaterializeRule
API provides a set of predefined rules that can be used to control the auto-materialization behavior of assets. These rules can be combined and configured to create an
AutoMaterializePolicy
that suits your needs. The available rules include: - `materialize_on_cron`: Materialize an asset partition based on a cron schedule. - `materialize_on_missing`: Materialize an asset partition if it has never been materialized before. - `materialize_on_parent_updated`: Materialize an asset partition if one of its parents has been updated more recently than it has. - `materialize_on_required_for_freshness`: Materialize an asset partition if it is required to satisfy a freshness policy. - `skip_on_backfill_in_progress`: Skip an asset’s partitions if targeted by an in-progress backfill. - `skip_on_not_all_parents_updated`: Skip materializing an asset or partition if not all of its parents have been updated. These rules can be added to or removed from an
AutoMaterializePolicy
using the
with_rules
and
without_rules
methods, respectively. If the predefined rules do not cover your specific use case, you might need to implement custom logic outside of the
AutoMaterializeRule
framework. This could involve creating custom sensors or orchestration logic that triggers materializations under the conditions you define. For the most up-to-date information and potential new features, including the ability to write custom auto-materialization rules, please refer to the latest Dagster documentation or inquire with the Dagster community.
2 Views