What’s the best way to handle conditional materialisation in the context of a partitioned asset? I.e. Some partitions need to be materialised, while some others don’t

In the context of a partitioned asset, conditional materialization can be managed using Dagster's auto materialization rules. These rules allow you to specify conditions under which an asset or asset partition should be materialized or skipped. Here are some of the rules that you can use to handle conditional materialization: 1. `AutoMaterializeRule.materialize_on_parent_updated`: This rule materializes an asset partition if one of its parents has been updated more recently than it has. 2. `AutoMaterializeRule.materialize_on_missing`: This rule materializes an asset partition if it has never been materialized before. 3. `AutoMaterializeRule.skip_on_parent_missing`: This rule skips materializing an asset partition if one of its parent asset partitions has never been materialized (for regular assets) or observed (for observable source assets). 4. `AutoMaterializeRule.skip_on_parent_outdated`: This rule skips materializing an asset partition if any of its parents has not incorporated the latest data from its ancestors. 5. `AutoMaterializeRule.skip_on_not_all_parents_updated`: This rule skips materializing an asset partition if any of its parents have not been updated since the asset's last materialization. It has a parameter

require_update_for_all_parent_partitions

which, if set to

true

, requires all upstream partitions in each upstream asset to be materialized since the downstream asset’s last materialization in order to update it. By combining these rules, you can create a policy that fits your specific needs for conditional materialization. For example, if you want to materialize a partition only if all upstream partitions are up to date, you could use

AutoMaterializeRule.skip_on_not_all_parents_updated

with

require_update_for_all_parent_partitions

set to

true

. Remember that these rules can be applied when defining your assets, allowing you to encode the logic for conditional materialization directly into your asset definitions. This way, Dagster can automatically handle the materialization conditions for you. For more detailed information on how to apply these rules, please refer to the documentation on auto materializing assets and the API documentation for

AutoMaterializeRule

. when materializing an unpartitioned asse... button Ability to create assets that can only b... button way to indicate that an asset or asset p... button Prevent Materialisation if all partition... button Thank you for your feedback!

@Scout When I say conditional materialisation, I refer to the

output_required

parameter of the asset decorator. Please try to answer my question again: What’s the best way to handle conditional materialisation in the context of a partitioned asset? I.e. Some partitions need to be materialised, while some others don’t

To handle conditional materialization in the context of a partitioned asset, where some partitions need to be materialized and others don't, you can use the

output_required

parameter in conjunction with the

yield

syntax within your asset definition. Here's a step-by-step guide on how to implement this: 1. Define your asset with `output_required=False`: This indicates that the asset does not always need to materialize its output. 2. Use the

yield

syntax within your asset function: Within the body of your asset function, you can conditionally yield an

Output

object based on your specific logic. If the conditions for materializing a partition are met, you yield an

Output

with the data for that partition. If not, you simply do not yield an

Output

for that partition. 3. Implement your conditional logic: Your asset function should include the logic that determines whether a partition should be materialized. This could be based on the presence of data, a date check, or any other condition relevant to your use case. Here's an example based on the documentation:

python from dagster import Output, asset  @asset(output_required=False) def conditional_asset(context):     partition_key = context.get_partition_key()     # Replace the following line with your condition for a specific partition     if should_materialize(partition_key):         data = compute_data_for_partition(partition_key)         yield Output(data)     # If the condition is not met, do not yield an Output

In this example,

should_materialize

is a placeholder for your condition-checking function, and

compute_data_for_partition

is a placeholder for the function that computes the data for a given partition. By using

output_required=False

and conditionally yielding

Output

, you can control which partitions of the asset are materialized on a per-execution basis. If your function does not yield an

Output

object for a particular partition, then no materialization event will be created for that partition, and downstream assets that depend on it will not be materialized.