Erik Goldman
04/30/2023, 9:06 PMDynamicOut
?
I have a case with ~50k items that are dynamically loaded from a database and need to be processed by ops that I map, but right now Dagster enumerates all 50k via DynamicOut
before processing the first one. this leads to super expensive startup times for the job, when I really want it to enumerate a few, process them, then enumerate moreQwame
05/01/2023, 4:57 AMdaniel
05/01/2023, 8:07 PMErik Goldman
05/02/2023, 11:07 AM