I have a partitioned job with two assets. The first asset retrieves a file to eventually split it into several if it is too big. The second asset performs a series of processes, however I would like each sub-file to be processed separately (for memory issues) so in separate runs. I think the most obvious solution would be to split this job into two jobs and add a sensor to detect if there is one or more sub-files per partition to know the number of RunRequest to trigger.
However, in my use it is rare that there are several files so I think it's a shame to have to separate the flow of these two assets. I wonder if it is not possible in the io_manager between the two assets, to take the first sub-file for processing (most of the time the only file) and for all other files to trigger a RunRequest on the second asset with the path of this sub-file as configuration. But I don't have the impression that this is feasible. Do you have any suggestions/solutions to achieve this? Thanks in advance