I've been struggling with this for a few days... I...
# ask-community
b
I've been struggling with this for a few days... I'm trying to use the IOManager for outputting pandas dataframes as parquet files - as shown in the screenshot. I'm loading the dataframe
df
in successfully (by passing the
df.head()
into the logger) but when I run this `context.resources.io_manager.handle_output(context, obj=df)`I get the error
AttributeError: 'NoneType' object has no attribute 'to_parquet'
. Which suggests that when I pass the dataframe into the io_manager, it becomes a NoneType. How does this work?
🤖 1
The weird thing is, although it's throwing out the error, it actually does output the parquet 🙄
o
hi @Barry Sun -- in general, you shouldn't be manually calling
<http://context.resources.io|context.resources.io>_manager.handle_output
(this will get automatically called by framework code to store the outputs of your ops). so if you have an op defined as
Copy code
@op
def do_some_output(context):
    # ...
    return some_pandas_dataframe
then the io manager will automatically be invoked on some_pandas_dataframe. So I think what's happening in your case is that you're manually calling this handle_output function inside the body of your op (which will succeed), and then not returning a value from your op, so then when dagster invokes the handle_output function automatically, it will be invoked on the return value of your op (which is None).
1
b
Hi Owen, thanks a lot! That makes a lot of sense 🙂 I'm not sure where I got that code from... must be somewhere obscure in the docs 😅
o
no problem! 🙂
113 Views