Dennis Gera
08/24/2023, 6:54 PMyear=2023/month=08/day=23/raw.parquet
?jamie
08/24/2023, 7:01 PMDennis Gera
08/24/2023, 7:19 PM@asset(
compute_kind="python",
name="raw_matrix_accounts_receivable",
key_prefix=["omie", "matrix"],
io_manager_key="parquet_s3_io_manager",
partitions_def=DailyPartitionsDefinition(start_date="2023/06/22", fmt="%Y/%m/%d", timezone=TZ, hour_offset=11),
)
def raw_matrix_accounts_receivable(context: OpExecutionContext, matrix_omie_client: OmieAPI) -> pd.DataFrame:
data: list = matrix_omie_client.list_all_accounts_receivable()
df = pd.json_normalize(data)
return df
I then have an IO manager similar to the PickledObjectS3IOManager(UPathIOManager)
here. I want to edit the method that determines the path so that it saves partitioned assets in a hive structure. Currently, it saves as 2023/08/23/raw.parquet
in the example in my first message.