I’ve been using opentelemetry traces to analyze the performance of some of our dagster jobs. Something that has surprised me is the number of SQL database connections that get opened and closed. A single op is opening and closing postgres DB connections 11 times! It appears that this has to do with how dagster’s run storage and event log storage work - they have context managers like

with self.connect(): …

which are used for almost all of their methods. Is there a way to trim this down, like pool the connections somehow? Those DB connection calls account for a surprisingly high fraction of our runtime and they seem unnecessary.

@Spencer Nelson appreciate this is a month back, but is this dagit issue related? https://github.com/dagster-io/dagster/issues/8466 I would love to resolve this issue, there is a fairly large chunk of Azure Log Analytics ingestion cost incurred from these queries too.

Yes, that looks very related. Dagster basically requires a connection pooling frontend like pgbouncer.