Philippe Laflamme

02/10/2023, 6:45 PM
Hi, I’m migrating a dagster instance from sqlite to postgresql and I’m comparing performance between the two. In general, they’re comparable, but one thing that is dramatically different is the latency between
events. In the historical sqlite-based runs, this is ~0.5s and with my Postgres setup it’s ~5s. I’ve made my postgres instance rather small on purpose to reduce cost and I’m going over the internet to reach it, so a lot of this can be explained by this, but a 10x performance difference still seems excessive. I’m seeing a lot of connection churn with many many queries to Postgres’ schema tables. Is there some logging I can turn on to get more details from Dagster’s perspective?
So my investigation lead to this potential improvement. I don’t think it accounts for everything, but it does make startup considerably faster.