When I use the

s3_pickle_io_manager

, yielding

DynamicOutput

is slow, it takes about one second to yield one. Even if I push the

DynamicOutput

to a list and return it all at once, it is still like this. My s3 service is a local s3 service built with MinIO, so there is no network problem. Is there a way to optimize it?

Hi @王昊! The actual machinery inside of the

s3_pickle_io_manager

is not particularly complex (basically just calling this function for each output). So my initial guess is that this is likely bumping into some bottleneck either in the pickling step or the s3 upload step (or both), rather than something in the Dagster machinery. How large are your output objects? It might also be worthwhile benchmarking your local s3 service in isolation to get a better idea of what performance you could expect

@owen I understand. I will try to conduct benchmark testing. The size of my output objects is approximately 108 bytes, which is very small. That's why I'm confused about the slow speed.

hm yeah -- I'm not familiar with MinIO but it would be surprising for it to take that long to upload such a small amount of data. let me know the results of your benchmarking and if that doesn't seem to be the source of the issue I can dig in to see if there might be a bottleneck on dagster's side