Has anyone else experience the Dagit UI becoming practically unusable when navigating to an Asset with a large number of partitions? For example, I have an Hourly Partitioned Asset starting from May 2020 - this translates to roughly 24,000 partitions. Whenever I select this Asset in the UI it locks up completely and generally fails to display any Asset details. I have the Asset configured this way because I want to process transactional data hourly from the start date of when we implemented our source system, using past partition dates as a way to backfill historical data. If having this many partitions is a current limitation of Dagster/Dagit - I'm curious what the recommended best practice for a situation like this would be. Thank you!

hi @AJ Floersch! what version of dagit/dagster are you on? we've made some improvements to the performance of the UI for cases such as that, which you might be able to take advantage of. Some of these perf improvements do require a database migration (

dagster instance migrate

)

Hey, @owen! I'm currently on 1.1.17 - I'll try bumping up to the latest and see if that helps (along with the instance migrate). Something else I noticed through all this was that backfilled partitions don't seem to display any data points on Partitioned Asset Plots. Running 1 partition does plot the metadata as expected, but running a series as a backfill does not plot any of the metadata for the Partition charts. Is that to be expected or do you think the above upgrade might solve this as well?

hm definitely not expected, and haven't heard of this reported before. it's possible that the upgrade will fix this but otherwise it would be great to file an issue

Sounds good! I will circle back with an Issue if the upgrades don't seem to resolve. Thank you for your continued support!

I have this same issue - I have a few assets partitioned anywhere between 20k and 40k, and it renders dagit (1.1.13) almost unusable.

Hey, @owen! Just wanted to circle back with a couple updates after upgrading to 1.1.19. For reference, I have two identical assets - the only difference being: • Asset (A) is set-up with daily partitions (1,025 total) and, • Asset (B) is set-up with hourly partitions (24,609 total). Following the upgrade, the overall UI performance did improve - however, Asset (B) does still result in regular Page Unresponsive warnings, though it does seem to push through them a bit quicker than before. The key difference I continue to notice is regarding the Partitioned Metadata Plots. Asset (A) is now successfully graphing plots for Backfilled runs (these hadn't been populating as I had noted in my previous message). However, Asset (B) still does not display Partitioned Metadata Plots - it does display Event Metadata Plots, but not the corresponding Partitioned ones. Nor does it provide any of the Materialization in last run details on the righthand Asset summary view that slides over from the Lineage view.