Created a channel to discuss topics related to async execution, following up from this message from @Daniel Gafni :

Hey guys! In this particular case it would be better to use multithreading instead (because this way we are not requesting the user to change the data loading code and because more code in the way may be blocking). As I already told @sandy, the async concurrent partitions loading implementation in the UPathIOManager was more a POC. Doing the same with multithreading should be more useful as it would not require any user code changes to get concurrent loading.

it would be better to use multithreading instead

yea i was thinking the same thing over the weekend after Sandy messaged me - threading was on the mind with the news that GIL optional PEP was just accepted. I think we still probably need some explicit opt-in to avoid “breaking changes” if users have IO-managers (or custom executors maybe) that happen to not be thread-safe for some reason.

The GIL shouldn't be a problem as I/O doesn't depend on it. Multiple threads can do I/O at the same time in Python. As to opt-in config parameter - I agree, definitely

Might get back to this. Running 18k dynamic steps in a single Run and they are being generated for the last 1.5 hour. The actual processing hasn’t started yet lol. I imagine same thing will happen with results collection

I wonder if it’s necessary to wait for all the fan-our steps creation to be over before starting to execute them? The steps are being created for 2 hours now, while they usually take around 30 minutes to execute.