Hi! Is there any way to skip downstream assets mat...
# ask-community
y
Hi! Is there any way to skip downstream assets materialization if the upstream asset has bad AssetObservation? We have dbt models covered with tests. We load them using load_assets_from_dbt_manifest and materialize them in the dbt job. After the dbt job finishes, we run another job that uses data from tables created by dbt models. But there might be a case when a model builds successfully, but tests for this model fail. In this case, we want to skip materializations for downstream assets (because they might use bad data).
y
when would you know the asset observation / data is bad?
also, do you want to skip downstream assets or fail the entire run instead?
y
Failing the entire run is exactly what we want to avoid. As for bad data, we have a dbt model and tests for this model (e.g., id column values must be unique). In this case, the model is built successfully (asset materialized), but the test for this model fails, and we get an asset observation with status failed. And in this case, we should not materialize downstream assets for this model asset.
also is there any way to get an upstream asset observations from one of the downstream assets (or at least get asset observations by asset key)?
y
also is there any way to get an upstream asset observations from one of the downstream assets (or at least get asset observations by asset key)?
there’s currently no built-in way to do so, but you can query the Dagster Instance directly to get the latest observation event. here’s a similar question: https://dagster.slack.com/archives/C01U954MEER/p1660073779130509?thread_ts=1660072660.277929&cid=C01U954MEER you can also the metadata (e.g. whether the status is failed) from that event, and then conditionally materialize the downstream assets.
cc @sandy regarding potential improvements about access to upstream asset observations and metadata
192 Views