I want to be able to define a job's flow so that if some assets fail it can still materialize downstream, but if some other assets fail, it will not.

We understand the need for flexibility in handling asset materialization, especially when dealing with complex data pipelines where not all asset dependencies are critical for the downstream assets to be materialized. In Dagster, assets typically become out-of-date when the data or code they depend on changes, and the system is designed to ensure that downstream assets are updated when upstream assets change. However, the current behavior in Dagster is that if an asset's upstream assets fail, the automatic materialization process for the given asset is blocked. This is because the system assumes that the downstream asset depends on the successful materialization of all its upstream assets. There has been a discussion in the community about enabling auto-materialization of an asset even if some of its upstream assets have failed. This would provide greater flexibility in pipelines where some failures are expected or where an asset is not strictly dependent on the success of all its upstream assets. A proposed solution was to introduce a user-configurable setting or flag that allows for the auto-materialization of assets even if some upstream assets have failed. However, this issue has been closed, and the current behavior remains that downstream assets will not materialize if upstream assets fail. In practice, this means that if you have a pipeline where an asset A depends on assets B and C, and B fails but C succeeds, asset A will not automatically materialize because of B's failure. If you require different behavior, you might need to implement custom logic to handle such scenarios. One potential workaround is to use sensors to listen for runs of a job to complete, regardless of success or failure, in order to kick off a downstream job. This approach can be used to approximate the desired behavior, although it may not be a perfect solution. For more detailed guidance and potential updates to this behavior, you may want to check the latest Dagster documentation or reach out to the community for any new developments or best practices.

@Scout How can I make an asset return failure on the code (provided I want to provide a failure case on my own)?

In Dagster, you can use asset checks to define custom failure conditions for your assets. An asset check allows you to specify logic that validates the data produced by an asset. If the check fails, you can return an

AssetCheckResult

with a failure condition. Here's an example of how you can define an asset check that returns a failure:

python from dagster import asset, AssetCheckResult, AssetCheckSeverity  @asset def my_asset():     # Your asset computation logic here     ...  @asset_check(asset=my_asset) def my_check(context, my_asset):     # Your custom check logic here     is_valid = ...  # Replace with your validation logic      if not is_valid:         return AssetCheckResult(             passed=False,             severity=AssetCheckSeverity.ERROR,  # Use WARN for non-blocking issues             metadata={"reason": "Validation failed due to ..."},  # Optional metadata         )     else:         return AssetCheckResult(passed=True)

In this example,

my_check

is an asset check associated with

my_asset

. The check performs some validation logic and returns an

AssetCheckResult

indicating whether the check passed or failed. If the check fails, you can set the severity to

ERROR

and provide additional metadata to describe the failure. Please note that the severity level (

ERROR

or

WARN

) does not impact the execution of the asset or downstream assets. It is used to determine how the check result appears in the UI. If you want to prevent downstream assets from materializing based on the result of an asset check, you would need to implement custom logic to handle this, as Dagster's default behavior is to block downstream materializations only if an upstream asset fails to compute, not if an asset check fails. For the most up-to-date information on asset checks and handling failures, please refer to the latest Dagster documentation.