Hi all! question regarding concurrent execution within an asset I need to read multiple files from external source system say from S3, I would like to build logic to read all the files concurrently within an asset, how can we achieve it?

I'm doing this with

asyncio

and

aioboto3

(a more user-friendly but less powefull tool is

s3fs.S3FileSystem

with

asynchronous=True

)

You could also model this as a dynamically-partitioned asset if you want visibility into the individual files as partitions, or as a graph-backed asset with a dynamic fan-out across the files.

Unfortunately, this will make concurrent loading impossible (for how). I would be really happy if Dagster started loading inputs into ops concurrently.

I'm not sure I follow

Sorry, actually I misinterpreted what you proposed. Fan-out processing would work of course. Dynamic partitions would require a separate asset to map them with AllPartitionsMapping. But if you want to load files concurrently inside just one op asyncio is your best choice.

Got it, that all makes sense