https://dagster.io/ logo
#ask-community
Title
# ask-community
v

veena

07/21/2023, 7:46 PM
Hi all! question regarding concurrent execution within an asset I need to read multiple files from external source system say from S3, I would like to build logic to read all the files concurrently within an asset, how can we achieve it?
dagster bot responded by community 1
d

Daniel Gafni

07/21/2023, 8:09 PM
I'm doing this with
asyncio
and
aioboto3
(a more user-friendly but less powefull tool is
s3fs.S3FileSystem
with
asynchronous=True
)
z

Zach

07/21/2023, 8:50 PM
You could also model this as a dynamically-partitioned asset if you want visibility into the individual files as partitions, or as a graph-backed asset with a dynamic fan-out across the files.
d

Daniel Gafni

07/21/2023, 8:58 PM
Unfortunately, this will make concurrent loading impossible (for how). I would be really happy if Dagster started loading inputs into ops concurrently.
z

Zach

07/21/2023, 9:02 PM
I'm not sure I follow
Maybe I misunderstood the question
d

Daniel Gafni

07/21/2023, 9:25 PM
Sorry, actually I misinterpreted what you proposed. Fan-out processing would work of course. Dynamic partitions would require a separate asset to map them with AllPartitionsMapping. But if you want to load files concurrently inside just one op asyncio is your best choice.
👍 1
z

Zach

07/21/2023, 9:38 PM
Got it, that all makes sense
2 Views