Hi team. How could I manipulate an <S3 IO manager>...
# ask-community
d
Hi team. How could I manipulate an S3 IO manager to save partitioned files in a hive structure
year=2023/month=08/day=23/raw.parquet
?
j
is the file path determined within the asset function like from the current date, or info in the data? or is it something you want to set as configuration?
d
this is my asset:
Copy code
@asset(
    compute_kind="python",
    name="raw_matrix_accounts_receivable",
    key_prefix=["omie", "matrix"],
    io_manager_key="parquet_s3_io_manager",
    partitions_def=DailyPartitionsDefinition(start_date="2023/06/22", fmt="%Y/%m/%d", timezone=TZ, hour_offset=11),
)
def raw_matrix_accounts_receivable(context: OpExecutionContext, matrix_omie_client: OmieAPI) -> pd.DataFrame:
    data: list = matrix_omie_client.list_all_accounts_receivable()
    df = pd.json_normalize(data)
    return df
I then have an IO manager similar to the
PickledObjectS3IOManager(UPathIOManager)
here. I want to edit the method that determines the path so that it saves partitioned assets in a hive structure. Currently, it saves as
2023/08/23/raw.parquet
in the example in my first message.