Philippe Laflamme01/27/2023, 5:11 AM
style for daily partitions and
for hourly. From my understanding, I can use the
or I can configure a custom IO manager to do this. If I use
, then users are exposed to how files are organized on disk (in
) which is not what I want: users should only care about “logical” partitions, not files on disk. If I use a custom IO manager, then it’s somewhat tedious since I have to define a separate one for each style of file organization on disk; which defies the purpose of having separate abstractions. It’s also somewhat brittle since the IO manager implementation receives partitions as string keys, which are already formatted (using the
). So the partition keys have to be parsed into `date`/`datetime` and then re-formated into a different style. Am I missing something here? What’s the approach for having finer control over how asset files are organized on disk without exposing this to
Peter Davidson01/27/2023, 8:03 AM
Philippe Laflamme01/27/2023, 2:48 PM
. So you have to do something like this
Which seems brittle to me. I’ll take a look at that second option you’re suggesting. That seems like a better way to change the IO manager’s behaviour. Thanks!
deals with figuring out which column to use to update / delete partitions from a table; so that was a good source for copy-pasting some code. Thanks again!