Hi All! First Time dagster user here. I'm porting an existing process to dagster, and using DynamicOutput to parallelize a bunch of sorting operations. Is there a way to have millions of DynamicOutputs without breaking the UI?

Hi @Edward Smith, I think this would fall outside the standard use cases for Dagster. Dagster is great for structuring your pipelines, getting observability, etc., but it’s not intended to compete with tools like Dask or Spark for high performance compute. I don’t know much about your usecase, but I think you could consider calling out to one of those tools from your solids. They’ll do the heavy-lifting that they’re good at, while you’ll still get the Dagster interface on top of them

Thanks, @johann. This process currently just uses the Python multiprocessing library in a single script, so it seems like this should 'fit' within dagster:

p = multiprocessing.Pool(30)
        <http://logger.info|logger.info>("Starting to push")
        with gzip.open(f"recommendations_parallel.json.gz", "wt") as fout:
            for num, result in enumerate(p.imap_unordered(save_recommendations, list(range(len(broadcaster_names))))):
                if num % 10000 == 0:
                    <http://logger.info|logger.info>(f"Sent {num}")
                    logger.debug(f"Sample Result: {json.dumps(result)}")
                if SAVE_RECOMMENDATIONS:
                    fout.write(json.dumps(result))
                    fout.write("\n")

Yeah I don’t have a strong opinion, but grouping the operations into one solid could be reasonable

That makes sense.... it does seem to me that the UI is the only limiting factor here, is that right?

We also write a number of events to your database for each solid. You might run into issues with number of concurrent connections

Well, the number of concurrent connections is limited by the number of threads

What executor would you be using?

I'm also thinking that maybe I could set a number of partitions in the solid config, and then, instead of yielding each individual item, I could yield N lists of items so that each list becomes a solid and there is a fixed number of lists.

You might find that the UI is the biggest limiting factor, but you would just be in uncharted territory and could certainly run in to something

Gotcha... I'll have my DynamicOutput be a list of X tuples instead of each tuple and I think that will resolve this issue.

With a million solids you’ll also pay the process spinup overhead for each solid

I can estimate the total number of tuples so that I wind up with a reasonable number of dynamic solids

Yeah I think finding the right way to group work is the way to go here

cool, thanks for the help!

You’re welcome!