I have asset (key: C) with 2 upstream asset dependencies (keys: A + B). I want to enable lazy materialization of asset A, meaning I want asset A to materialization only if the downstream asset C needs to materialize and have it's freshness policy met. However, I do NOT want asset B to need to be fresh. Asset B is an optional dependency, and it does not need to be fresh in order for asset C to meet it's requirements. Is this possible in dagster?

In Dagster, you can define assets with various materialization policies, including lazy materialization. However, the documentation does not explicitly describe a built-in mechanism to selectively apply freshness requirements to only certain upstream assets while treating others as optional in the context of lazy materialization. The closest feature mentioned in the documentation is the

AutoMaterializePolicy.lazy()

which can be used to delay the materialization of an asset until its downstream assets require it. However, this policy does not provide a direct way to specify that only certain upstream assets need to be fresh for the downstream asset to materialize. One potential approach to achieve a similar result would be to use the

AutoMaterializePolicy

with custom rules that define the conditions under which an asset should be materialized. For example, you could use

AutoMaterializeRule.skip_on_not_all_parents_updated()

to skip materialization if not all parents have been updated, but this would apply to all upstream assets, not just the ones you want to be fresh. Another approach mentioned in the discussions is to modify the asset definition to yield optional outputs based on the materialization state of its dependencies. This would allow you to check if the required dependencies are materialized and only yield the output if so, effectively making some dependencies optional. Since the documentation does not provide a clear example or feature that exactly matches your requirements, it might be necessary to implement custom logic within your asset definitions to check the freshness of specific upstream assets and decide whether to materialize the downstream asset accordingly. If you need further guidance on how to implement this behavior, it would be best to refer to the latest Dagster documentation or reach out to the Dagster community for more specific use cases and potential workarounds.