I see a ton of these awesome dataframe constraints built out

Question

I see a ton of these awesome dataframe constraints built out in the dagster pandas library but I m unclear on how to properly leverage them in a real world pipeline <https github com dagster io dagster blob master python modules libraries dagster pandas dagster pandas tests test constraints py L37>

max · Answer

< abhi> are you on a plane slightly smiling face

dwall · Answer

so it looks like my DataFrame is failing a type check but here is the full traceback ```dagster core engine engine inprocess DagsterTypeCheckDidNotPass Type check failed for step output test dataframe of type TestDataFrame File Users dwall local share virtualenvs dataland dagster Z2VR7MFq lib python3 8 site packages dagster core engine engine inprocess py line 274 in dagster event sequence for step for step event in check generator core dagster event sequence for step step context File Users dwall local share virtualenvs dataland dagster Z2VR7MFq lib python3 8 site packages dagster core engine engine inprocess py line 556 in core dagster event sequence for step for evt in create step events for output step context user event File Users dwall local share virtualenvs dataland dagster Z2VR7MFq lib python3 8 site packages dagster core engine engine inprocess py line 581 in create step events for output for output event in type checked step output event sequence step context output File Users dwall local share virtualenvs dataland dagster Z2VR7MFq lib python3 8 site packages dagster core engine engine inprocess py line 502 in type checked step output event sequence raise DagsterTypeCheckDidNotPass ```

dwall · Answer

am I missing something or is it not telling me which type check failed

max · Answer

hm is that in a test using execute pipeline

dwall · Answer

yes

dwall · Answer

but traceback looks identical in dagit as well

max · Answer

you should be able to look at `metadata entries` on the exception object

max · Answer

and those should display in dagit are they not

dwall · Answer

let me take another look

dwall · Answer

ah okay they appear before the exception

dwall · Answer

```2020 02 14 14 27 06 dagster DEBUG test gsheet client 5668136f 0f45 46ec 9942 be014cbcae95 STEP OUTPUT Yielded output test dataframe of type TestDataFrame Warning Type check failed event specific data = intermediate materialization null step output handle gsheet to dataframe compute test dataframe type check data false test dataframe Violated ColumnTypeConstraint Column dtype must be datetime64 ns for Column Name date ```

dwall · Answer

nice

abhi · Answer

Glad everything got sorted out here When you get something working in your investigations I would love to chat about your thoughts on the API and improvements we could make there

dwall · Answer

< max> btw the metadata attached to the type check output is NOT showing up in dagit

dwall · Answer

I see this in console ```2020 02 14 15 32 21 dagster DEBUG test gsheet client dfed1330 31aa 45fc ade9 1fe05f5af8bb STEP OUTPUT Yielded output test dataframe of type TestDataFrame Warning Type check failed event specific data = intermediate materialization null step output handle gsheet to dataframe compute test dataframe type check data false test dataframe Violated UniqueColumnConstraint Column must be unique for Column Name id The offending index row values are the following date amount id notes approved category n3 2020 04 01 600 34 345 another string True type 1 ``` and this in Dagit ```Output Yielded output test dataframe of type TestDataFrame Warning Type check failed ```

max · Answer

that s not intended behavior

max · Answer

thanks for this

max · Answer

will have a diff up shortly

dwall · Answer

thumbsup thumbsup