Hi all I would like to start a discussion about Pandas dataframe model validation. Are there any useful libraries that integrate with Dagster that help to organize dataframes modifications? Right now we are integrating Pandera in our project, but I'm curious if there are any libraries that you are using for such case that I may miss.

Hi, i find it useful to mention https://github.com/dagster-io/dagster/discussions/15880 here. Join discussion, folks 🎉

<!channel> we've put together a design proposal for "Asset Checks": https://github.com/dagster-io/dagster/discussions/15880. Here's the summary:

Dagster allows you to define and execute data quality checks on your software-defined assets. Each asset check verifies some property of a data asset, e.g. that is has no null values in a particular column.

When viewing an asset in Dagster’s UI, you can see all of its checks, and whether they’ve passed, failed, or haven’t run. When launching a run to execute an asset, by default its checks will also be executed. Checks can also be executed on their own, independent of asset materializations.

By setting their severity level to ERROR, you can specify that your checks impact control flow, i.e. only materialize downstream assets if the checks on the upstream assets succeed.

We would appreciate any and all feedback on the proposal. (This is a re-post of what @Alex Orlovskyi already discovered, with some additional context and an <!channel> mention for wider distribution).

<!channel> we shipped a very early version of asset check APIs and UI in 1.4.11. We’d love for you to try them out and share feedback! Directions: https://github.com/dagster-io/dagster/discussions/16266

Is anyone working on a integration with https://github.com/datafold/data-diff ?

<!channel> we shipped an experimental integration of the asset check API's with dbt. As of

dagster-dbt>=0.20.13

, dbt tests can now be modeled as Dagster asset checks. For more details and requirements for setup, see: https://github.com/dagster-io/dagster/discussions/16527.

👋 Hey folks, We’re looking to push our data quality story in the upcoming quarter. One of the questions I had was how much do we expect the asset_check stuff to change? Obviously it’s experimental so some occasional API breaks are expected, but we’d still like to avoid signficant refactoring if we go down this route. Do people have any thoughts on how if we should consider getting into asset_checks early, and if so if there are any conventions/considerations that may not be documented that may make it a bit easier to roll with whatever changes may occur?

In addition to DBT support, last week we shipped • a new execute button for individual checks (intended for when you’re developing a check and want to rerun it without re materializing) • Support for checks on

@graph_asset

and

@graph_multi_asset

. These can be used to implement checks that block materializations. (discussion) To try them out: https://github.com/dagster-io/dagster/discussions/16266

For me, AssetCheckSeverity.ERROR is not performing as expected. I am still getting downstream assets materializing after an error-failure.

Unrelated - Do we have an estimate on when asset-checks per-partition will be explored/implemented?

Breaking change: our 1.5 release planned for this Friday the 29th will change

AssetCheckResult(success=…)

to

AssetCheckResult(passed=…)

. We’re also going to make an internal change, so to use checks with Dagster Cloud you’ll need dagster version 1.5

Hello, I'm currently using checks via the dbt integration. Our dbt repo is separate from Dagster so we do a few automated CI things to sync the manifest file over. Last night dbt PR was merged and things sync'd over to Dagster as expected, bug Dagster code location failed due to a check error which I can't pin point root cause.

dagster._core.errors.DagsterInvalidDefinitionError: Duplicate check specs: {(AssetKey(['ANALYTICS', 'my_model_name']), 'assert_[some_name]_matches'): 2}

I'm able to find "my_model_name" in dbt, but I don't see any duplicate tests in schema file. I don't know how it's creating/getting the

assert[...]

part and I tried searching for all and substrings in the dbt repo but still nothing. For now, I've disabled the checks in

dbt_project.yml

inorder to get Dagster up and running again. Any ideas?

Not sure if it's related to asset checks but I'm getting a bunch of GraphQL errors like these

Not sure if it's the correct channel, but can you apply Asset Checks for Source Assets? Let's say I have a very large table on Postgres, and I want to run health check on it every hour (Freshness check). Is this supported yet? Or is it in the pipeline?

with

dagster-dbt 0.21.1

we can now run dbt checks individually, is there a difference between yielding from

build

vs. separating into

run

and

test

streams?

@johann is build_asset_with_blocking_check / build_blocking_asset_check released yet?

Hello all, really enjoying the asset checks and it is definitely going to make life easier. However, I'm encountering an issue right now. Here's my code for the asset and asset check:

@asset(compute_kind="python")
def my_data(context) -> None:
    data = wr.s3.read_parquet(cfg.s3_path)
    connection = duckdb.connect(database=os.fspath(duckdb_database_path), read_only=False)
    connection.execute("create schema if not exists database")
    connection.execute(
        "create or replace table database.my_data as select * from data"
    )


@asset_check(asset=my_data, description="Check that my asset has enough rows")
def my_asset_has_enough_rows() -> AssetCheckResult:
    connection = duckdb.connect(database=os.fspath(duckdb_database_path), read_only=False)
    df = connection.execute("select * from database.my_data").fetchdf()
    num_rows = df.shape[0]
    return AssetCheckResult(passed=num_rows > 5, metadata={"num_rows": num_rows})

I read in a parquet file from S3 and load it into a duckdb file. My asset check queries the duckdb file and checks to see if

num_rows > 5

. The issue is that the asset check errors with the following:

dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/Users/.../virtualenvs/.../lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 247, in execute
    event_or_none = next(step_iter)
  File "/Users/.../virtualenvs/.../lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 357, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
  File "/Users/.../virtualenvs/.../lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 174, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)

It seems like this is caused because of the multiprocessing (running the asset check while executing downstream assets). My downstream assets are DBT assets that also query the duckdb. I saw that there is a way to include the asset check in the @asset declaration, but I like having them separate. Any thoughts? It would be nice to only begin materializing downstream assets until after the asset_check execution succeeds. Note: this is only an issue if the asset_check queries the duckdb file. If I just pass return

data

from the asset and pass it into the asset_check directly, the check works fine. So this is a working alternative but I like the idea that the asset_check could query the duckdb instead of passing dataframes around.

I'm trying to add checks with the new @asset_checks and have three questions. 1. is there no way to reference multiple assets with one check? is something like this not possible? @asset_check(assets=[asset1, asset2, asset3] 2. is there a way to find the asset_key of the referenced asset in a check? something like this: EventRecordsFilter(DagsterEventType.ASSET_MATERIALIZATION, asset_key=context.asset_key) 3. is there nothing like load_asset_checks_from_modules() to create the Definition object? Looks like I need to manually specify all checks?

Is it already clear when the asset checks DBT integration will support running the tests on their own dadence?

I'm experimenting with the dbt asset checks functionality. I'm using

DagsterDbtTranslatorSettings(enable_asset_checks=True)

which seems to work fine - I can run asset checks from UI. However, materialising the asset does not automatically trigger the asset checks. The asset body is just:

dbt.cli(["build"], context=context).stream()

but the asset checks on the GUI are show 'skipped' (and the checks are not run, based on the logs). What's going wrong? Slack Conversation

I noticed there’s already historical logging of asset check results, in the future, will we be able to plot this as well? (Similar to how metadata may be plotted!)

Has anyone succeeded adding checks to current assets. My attempt at this (and it failed): https://github.com/dagster-io/dagster/issues/17469

https://github.com/dagster-io/dagster/issues/17560 and https://github.com/dagster-io/dagster/issues/17372 around DBT and asset checks area still unclear for me how to handle them

Does anyone know if the

multi_asset_check

functionality from the initial discussion has been implemented or cancelled? I can't find any reference to it outside of that post.

Hello all, wondering if anyone else has run into a similar issue. I am able to successfully call my asset checks using the following Definitions:

defs = Definitions(
    assets=[assets.my_asset],
    asset_checks=[assets.my_asset_has_enough_rows],
)

However, it seems that when I use the function,

load_assets_from_package_module

, it is not loading the asset check:

defs = Definitions(
    assets=load_assets_from_package_module(source, group_name='test', key_prefix='test'),
    asset_checks=[assets.my_asset_has_enough_rows],
)

In this example,

assets.py

is in the sources subdirectory. All the assets show up, just no asset checks including DBT tests. Anyone run into this?

Hey. I started using asset checks in my dbt project. It's such an awesome feature, this is something we struggled a lot with in airflow projects. We had use a few tricks to run tests in our DAGs that would prevent the materialization of downstream models if tests failed. It's great to see first class support for dbt! However, I noticed that severity levels for tests in a dbt project are not mapped to those of the produced data asset. Is there a way to use the severity level in dbt config when translating tests to asset tests?