The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

I built a simple pipeline to apply an xgboost model to a new dataset and I'm getting issues with loading the pickled model object
```_pickle.UnpicklingError: NEWOBJ class argument isn't a type object ```
Does anyone know why dagster isn't recognizing the object?

would you mind sharing more of the traceback?

```_pickle.UnpicklingError: NEWOBJ class argument isn't a type object
  File "/usr/local/lib/python3.7/site-packages/dagster/core/errors.py", line 174, in user_code_error_boundary
    yield
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/execute.py", line 689, in _user_event_sequence_for_step_compute_fn
    for event in gen:
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/compute.py", line 93, in _execute_core_compute
    for step_output in _yield_compute_results(compute_context, inputs, compute_fn):
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/compute.py", line 64, in _yield_compute_results
    for event in user_event_sequence:
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/decorators.py", line 421, in compute
    result = fn(context, **kwargs)
  File "/opt/dagster/pipelines/dagster_examples/xgboost_test.py", line 20, in apply_model
    model = pickle.load(infile)```

Here's the pipe (it's just a demo) <https://github.com/CodeForPhilly/dsdk-dagster/blob/master/pipelines/dagster_examples/xgboost_test.py>

what is writing `dagster_examples/model_pickle.dat`?

and what is the python type of that pickled object?

this script works in jupyter also, I can load this object in a notebook and apply without issues

my first suspicion would be that maybe you're using a different version of python/pickle in jupyter than with dagster

i would be curious to see if using `XGBClassifier.dump_model` and `XGBClassified.load_model` resolved this

well, it depends what environment you're running dagster in. you could run `python --version` and `which python` in that environment and compare it against `jupyter kernelspec list`

and in the notebook you're just calling `pickle.dump`?

could it be this? <https://discuss.xgboost.ai/t/xgbclassifier-un-pickling-fails-if-scikit-learn-is-not-istalled/1520>

that fixed it!  On to the new error :laughing: