https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Philippe Laflamme

02/10/2023, 6:45 PM
Hi, I’m migrating a dagster instance from sqlite to postgresql and I’m comparing performance between the two. In general, they’re comparable, but one thing that is dramatically different is the latency between
STEP_WORKER_STARTING
and
STEP_WORKER_STARTED
events. In the historical sqlite-based runs, this is ~0.5s and with my Postgres setup it’s ~5s. I’ve made my postgres instance rather small on purpose to reduce cost and I’m going over the internet to reach it, so a lot of this can be explained by this, but a 10x performance difference still seems excessive. I’m seeing a lot of connection churn with many many queries to Postgres’ schema tables. Is there some logging I can turn on to get more details from Dagster’s perspective?
So my investigation lead to this potential improvement. I don’t think it accounts for everything, but it does make startup considerably faster.
2 Views