https://dagster.io/ logo
#ask-ai
Title
# ask-ai
f

Fernando Cagua

02/29/2024, 8:29 PM
What’s the best way to handle conditional materialisation in the context of a partitioned asset? I.e. Some partitions need to be materialised, while some others don’t
s

Scout

02/29/2024, 8:29 PM
In the context of a partitioned asset, conditional materialization can be managed using Dagster's auto materialization rules. These rules allow you to specify conditions under which an asset or asset partition should be materialized or skipped. Here are some of the rules that you can use to handle conditional materialization: 1. `AutoMaterializeRule.materialize_on_parent_updated`: This rule materializes an asset partition if one of its parents has been updated more recently than it has. 2. `AutoMaterializeRule.materialize_on_missing`: This rule materializes an asset partition if it has never been materialized before. 3. `AutoMaterializeRule.skip_on_parent_missing`: This rule skips materializing an asset partition if one of its parent asset partitions has never been materialized (for regular assets) or observed (for observable source assets). 4. `AutoMaterializeRule.skip_on_parent_outdated`: This rule skips materializing an asset partition if any of its parents has not incorporated the latest data from its ancestors. 5. `AutoMaterializeRule.skip_on_not_all_parents_updated`: This rule skips materializing an asset partition if any of its parents have not been updated since the asset's last materialization. It has a parameter
require_update_for_all_parent_partitions
which, if set to
true
, requires all upstream partitions in each upstream asset to be materialized since the downstream asset’s last materialization in order to update it. By combining these rules, you can create a policy that fits your specific needs for conditional materialization. For example, if you want to materialize a partition only if all upstream partitions are up to date, you could use
AutoMaterializeRule.skip_on_not_all_parents_updated
with
require_update_for_all_parent_partitions
set to
true
. Remember that these rules can be applied when defining your assets, allowing you to encode the logic for conditional materialization directly into your asset definitions. This way, Dagster can automatically handle the materialization conditions for you. For more detailed information on how to apply these rules, please refer to the documentation on auto materializing assets and the API documentation for
AutoMaterializeRule
. when materializing an unpartitioned asse... button Ability to create assets that can only b... button way to indicate that an asset or asset p... button Prevent Materialisation if all partition... button Thank you for your feedback!
f

Fernando Cagua

02/29/2024, 8:31 PM
@Scout When I say conditional materialisation, I refer to the
output_required
parameter of the asset decorator. Please try to answer my question again: What’s the best way to handle conditional materialisation in the context of a partitioned asset? I.e. Some partitions need to be materialised, while some others don’t
s

Scout

02/29/2024, 8:31 PM
To handle conditional materialization in the context of a partitioned asset, where some partitions need to be materialized and others don't, you can use the
output_required
parameter in conjunction with the
yield
syntax within your asset definition. Here's a step-by-step guide on how to implement this: 1. Define your asset with `output_required=False`: This indicates that the asset does not always need to materialize its output. 2. Use the
yield
syntax within your asset function:
Within the body of your asset function, you can conditionally yield an
Output
object based on your specific logic. If the conditions for materializing a partition are met, you yield an
Output
with the data for that partition. If not, you simply do not yield an
Output
for that partition. 3. Implement your conditional logic: Your asset function should include the logic that determines whether a partition should be materialized. This could be based on the presence of data, a date check, or any other condition relevant to your use case. Here's an example based on the documentation:
Copy code
python from dagster import Output, asset  @asset(output_required=False) def conditional_asset(context):     partition_key = context.get_partition_key()     # Replace the following line with your condition for a specific partition     if should_materialize(partition_key):         data = compute_data_for_partition(partition_key)         yield Output(data)     # If the condition is not met, do not yield an Output
In this example,
should_materialize
is a placeholder for your condition-checking function, and
compute_data_for_partition
is a placeholder for the function that computes the data for a given partition. By using
output_required=False
and conditionally yielding
Output
, you can control which partitions of the asset are materialized on a per-execution basis. If your function does not yield an
Output
object for a particular partition, then no materialization event will be created for that partition, and downstream assets that depend on it will not be materialized.
2 Views