with the error `Type check failed for step input df test exp

Question

with the error `Type check failed for step input df test expected type DataFrame ` because i clearly have not figured out how to supply the two assets to this job I thought of using `input values` and assets as inputs as I might find myself using different files later But is this the right way I am doing it at the job level because the same graph might be used in a different context training vs test vs production set for example but then how do i set `df train` and `df test`to the appropriate assets

jamie · Answer

What happens if you add type hints to your assets ``` asset def train dataset gt pd DataFrame train data str = data train csv return read data train data asset def test dataset gt pd DataFrame test data str = data test csv return read data test data ```

Rahul Dave · Answer

Same error I think input values is not expecting an `AssetKey`

Rahul Dave · Answer

`dagster core errors DagsterTypeCheckDidNotPass Type check failed for step input df test expected type DataFrame Description Value of type lt class dagster core definitions events AssetKey gt failed type check for Dagster type DataFrame expected value to be of Python type pandas core frame DataFrame `

jamie · Answer

ah yeah silly oversight on my part you ll also need to convert the two assets to ops in order to use them with other ops

Rahul Dave · Answer

yeah am trying that now i think there is no way to be part assets unless we become asset backed graphs And we cant do that for now

Rahul Dave · Answer

I finally managed to get it to work with this

Rahul Dave · Answer

``` resource def current training data init context return data train csv resource def current testing data init context return data test csv def read data data str return pd read csv data op required resource keys= training data def read train data context gt pd DataFrame return read data context resources training data op required resource keys= testing data def read test data context gt pd DataFrame test data str = data test csv return read data context resources testing data encoder op = define dagstermill op name= encoder op notebook path=file relative path file notebooks encoder ipynb output notebook name= output encoder outs= encoders Out dict ins= df train In pd DataFrame df test In pd DataFrame graph out = encoders GraphOut def encoder graph df train = read train data df test = read test data encoders = encoder op df test df train return encoders local encoder job = encoder graph to job name= local encoder job resource defs= output notebook io manager local output notebook io manager training data current training data testing data current testing data ```

Rahul Dave · Answer

Any thoughts criticisms I now need to hook up the output to an appropriate iomanager its using the default so i can use the output elsewhere or store it permanently

jamie · Answer

this looks mostly good I m a bit confused about the duplication of the testing training resources since the same paths are variables in the corresponding ops and the need for the resources in general is there more going on in those resources than makes sense for a code snippet

Rahul Dave · Answer

I want to use those resources in other pipelines For example a training job will take the encoders output here and the training data to first transform the data and then train on it This will store a model somewhere Now on test data or completely new data I want to run the encoders and transform the data which is part of the same pipeline as before but replace the training by prediction using the model

jamie · Answer

cool sounds good

Rahul Dave · Answer

The other thing I am trying to achieve unsuccessfully with this code since read data is included twice is to say what if I replace the csv by parquet The proper solution to this I think configurable io managers but I have to figure out how to do it

jamie · Answer

you could have your resource read in the file and return the data as a dataframe then you could write some custom logic for how to load if the file is csv vs parquet

Rahul Dave · Answer

And hook it up to a run config to dispatch based on file ending or custom flag say If I understand you right you are saying io managers are really for data transport and to think of data type orthogonally