Hi again y all I m building a pipeline that will need to dow

Question

Hi again y all I m building a pipeline that will need to download over a hundred files I m wondering if I should handle the concurrent downloads using dagster s `multiprocess executor` or if I m better off using something like aiohttp

Auster Cid · Answer

I m guessing I ll have a huge execution overhead using the executor but maybe there s a better way I m not aware of Anyway sorry for the noob question I m still going through the docs

Lyle · Answer

i m new to dagster myself but it seems like aiohttp would be advantageous here i m with you on the overhead bit multiproc seems like overkill for just downloading files maybe do both and timeit ツ

alex · Answer

ya I would lean aiohttp or something designed to do the task well the multi process executor is currently not optimized for anything other than doing each step execution in a clean process for isolation