https://dagster.io/ logo
#ask-community
Title
# ask-community
u

王昊

05/22/2023, 8:47 AM
When I use the
s3_pickle_io_manager
, yielding
DynamicOutput
is slow, it takes about one second to yield one. Even if I push the
DynamicOutput
to a list and return it all at once, it is still like this. My s3 service is a local s3 service built with MinIO, so there is no network problem. Is there a way to optimize it?
o

owen

05/22/2023, 7:57 PM
Hi @王昊! The actual machinery inside of the
s3_pickle_io_manager
is not particularly complex (basically just calling this function for each output). So my initial guess is that this is likely bumping into some bottleneck either in the pickling step or the s3 upload step (or both), rather than something in the Dagster machinery. How large are your output objects? It might also be worthwhile benchmarking your local s3 service in isolation to get a better idea of what performance you could expect
u

王昊

05/23/2023, 3:08 AM
@owen I understand. I will try to conduct benchmark testing. The size of my output objects is approximately 108 bytes, which is very small. That's why I'm confused about the slow speed.
o

owen

05/23/2023, 3:59 PM
hm yeah -- I'm not familiar with MinIO but it would be surprising for it to take that long to upload such a small amount of data. let me know the results of your benchmarking and if that doesn't seem to be the source of the issue I can dig in to see if there might be a bottleneck on dagster's side
2 Views