Hi all, thanks for the help as always! I'm having trouble reading in raw files where the exact file names are unknown at runtime. This is due to file names having variations from manual inputs or system changes - but usually there's some consistent strings that might be used - e.g. to indicate the month of the file. What is the best way to incorporate this into my IOManager? I'm currently playing around with MonthlyPartitions to read these files in as an SDA, as they do follow a monthly pattern. The issue is the naming convention of the raw files isn't very consistent. At the moment, I'm looking at using the context.asset_partition_key and context.step_key as regex strings to search for the appropriate file.

Hi Barry. This is an interesting problem--I think that the easiest way to handle the unknown file name will probably be to define your own custom IO manager. Just based on the example here, you could define your own

path_for_partition

method and use the partition key to determine the exact path for the file, which sounds like something you might already be doing

Thanks Claire! I did manage to get it working yesterday 😄