Dagster-pandas StrictColumnsConstraint is not strict. Bug?
If I provide extraneous (ie non existing) columns to StrictColumnsConstraint the test will pass (which is incorrect). Leaving off a column will not (correct behavior).
Also it would be useful to have more information in the stacktrace/log when a test fails. Now it only says: “Type check failed for step output “df” - expected type “VerrichtingCodesFrame”.” It would be hard to find out why the typecheck fails, esp. when one has more constraints. Ideally it would say which constraint was violated and where the error occurred.
FYI this is my type definition
VerrichtingCodesFrame = create_dagster_pandas_dataframe_type(
name="VerrichtingCodesFrame",
dataframe_constraints=[
RowCountConstraint(51),
StrictColumnsConstraint(
['lokale_code', 'fenac_code', 'in_gebruik', 'lokale_omschrijving',
'domein', 'gebied', 'voorkeurgebied', 'fenac_omschrijving',
'toelichting', 'normduur']
)]
)
I’m running Dagster 0.13.2