I have a partitioned asset that depends on an unpartitioned asset. When the unpartitioned asset is materialized it automatically makes all partitions of the downstream asset stale. How can I avoid this?

We understand that managing dependencies between partitioned and unpartitioned assets can be challenging, especially when it comes to materialization and freshness. According to the documentation, when an unpartitioned asset is upstream of a partitioned asset, Dagster assumes that each partition of the downstream asset depends on the unpartitioned asset. This means that when the unpartitioned asset is updated, all partitions of the downstream asset are considered stale. However, there are ways to customize this behavior using auto-materialization rules. For instance, you can use the

_skip_on_not_all_parents_updated

rule with the

require_update_for_all_parent_partitions

parameter set to

False

. This rule allows you to skip materializing an asset partition if not all of its parents have been updated since the asset’s last materialization. By default, this parameter is

False

, which means that it requires at least one upstream partition in each upstream asset to be materialized since the downstream asset’s last materialization in order to update it. Another rule that might be relevant is

_skip_on_backfill_in_progress

, which can skip an asset’s partitions if targeted by an in-progress backfill. The

all_partitions

parameter can be set to

True

to skip all partitions of the asset being backfilled, regardless of whether the specific partition is targeted by a backfill. If these rules do not fit your use case, you might consider implementing custom logic within your asset definitions to handle the staleness and materialization conditions more explicitly. For example, within the asset computation, you could check the materialization status of the upstream asset and decide whether to proceed with the computation for the partitioned asset. For more advanced scenarios, you might need to look into custom partition mappings or sensors that trigger materialization based on more complex conditions. However, as of the last update from the documentation, defining custom

PartitionMappings

is not supported due to performance constraints when calling out to custom user code defined within

PartitionMappings

. If you are encountering specific issues or need further customization, it might be helpful to look at the discussions and issues on the Dagster GitHub repository, where similar challenges have been discussed, and potential solutions or workarounds might be available.