recently my team has been using dagster to orchestrate some large analytics runs, and we've noticed when a job run has thousands to tens of thousands of steps the UI performance takes a pretty hard hit when viewing the run. UI performance will continue to suffer after navigating away from the run (I assume step state is still being retrieved in the background). The performance is fine if the user avoids loading any runs with thousands of steps. This is using the a remote postgres storage backend on RDS, so I understand there's probably some unavoidable latency retrieving state for that many steps, just thought I'd mention it in case it's useful.

@prha @alex

certainly more improvements to be made tuning things for runs of that magnitude

We also notice the same on bigger systems. Two table on postgres quickly grew into two digits GBs. We only run postgres on a single pod. Upgrading memory and CPU helped a bit, but not for long.