I built a simple pipeline to apply an xgboost model to a new

Question

I built a simple pipeline to apply an xgboost model to a new dataset and I m getting issues with loading the pickled model object ``` pickle UnpicklingError NEWOBJ class argument isn t a type object ``` Does anyone know why dagster isn t recognizing the object

max · Answer

would you mind sharing more of the traceback

Marieke · Answer

sure

Marieke · Answer

``` pickle UnpicklingError NEWOBJ class argument isn t a type object File usr local lib python3 7 site packages dagster core errors py line 174 in user code error boundary yield File usr local lib python3 7 site packages dagster core execution plan execute py line 689 in user event sequence for step compute fn for event in gen File usr local lib python3 7 site packages dagster core execution plan compute py line 93 in execute core compute for step output in yield compute results compute context inputs compute fn File usr local lib python3 7 site packages dagster core execution plan compute py line 64 in yield compute results for event in user event sequence File usr local lib python3 7 site packages dagster core definitions decorators py line 421 in compute result = fn context kwargs File opt dagster pipelines dagster examples xgboost test py line 20 in apply model model = pickle load infile ```

Marieke · Answer

Here s the pipe it s just a demo <https github com CodeForPhilly dsdk dagster blob master pipelines dagster examples xgboost test py>

max · Answer

what is writing `dagster examples model pickle dat`

Marieke · Answer

it s a saved object in the repo

Marieke · Answer

I built it in jupyter

max · Answer

and what is the python type of that pickled object

Marieke · Answer

xgbclassifier

max · Answer

hmm ok

Marieke · Answer

this script works in jupyter also I can load this object in a notebook and apply without issues

max · Answer

my first suspicion would be that maybe you re using a different version of python pickle in jupyter than with dagster

Marieke · Answer

is it that it s using python slim

Marieke · Answer

dagster

max · Answer

i would be curious to see if using `XGBClassifier dump model` and `XGBClassified load model` resolved this

Marieke · Answer

good point lemme try that

max · Answer

well it depends what environment you re running dagster in you could run `python version` and `which python` in that environment and compare it against `jupyter kernelspec list`

Marieke · Answer

they re both on 3 7

max · Answer

and in the notebook you re just calling `pickle dump`

Marieke · Answer

yeah

Marieke · Answer

I also tried dill

max · Answer

could it be this <https discuss xgboost ai t xgbclassifier un pickling fails if scikit learn is not istalled 1520>

Marieke · Answer

OHHH

Marieke · Answer

lemme see

Marieke · Answer

that fixed it On to the new error laughing

Marieke · Answer

thank you

max · Answer

np