Database creation performance drop going from pg 14 to pg 15+

From: "Mahdi Bahrami" <pgsql(at)mahdibm(dot)com>
To: pgsql-performance(at)lists(dot)postgresql(dot)org
Subject: Database creation performance drop going from pg 14 to pg 15+
Date: 2025-05-15 16:28:50
Message-ID: 9649fa2b-4664-4ff2-9da1-17d9b89ed44a@app.fastmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

This is about a performance issue / unexpected behavior that I found when investigating https://github.com/vapor/postgres-nio/issues/506.
The related message to this thread is
https://github.com/vapor/postgres-nio/issues/506#issuecomment-2883158951.
As a TL;DR about that issue, a user reported a performance problem when going from postgres 13 to any postgres' higher, and upon my investigations, the main issue turned out to be the fact that pg 13 docker images default to md5 auth, and postgres 14+ default to scram-sha-256. The library had an unoptimized scram-sha-256 implementation, which has since been resolved, making the auth almost as fast as md5, so the main problem is resolved.

The other problem is what I have questions about. It's a performance regression noticed in pg 15 and higher, with pg 13 and 14 being faster than 15+, that i mentioned in https://github.com/vapor/postgres-nio/issues/506#issuecomment-2883158951.

To set things up, the project used to reproduce all these is https://github.com/finestructure/pg-perf-regression.git, branch `check-fixes`. You can run the benchmark using the `./run-perf-test.sh` script (a simple one). But you need to have a recent Swift installed (see https://swift.org, use swiftly) and docker running. Then try running the script using
`./run-perf-test.sh 14 test1`
with `14` being the postgres version.

The problem can be seen in the 2 captures attached to this email. Each capture contains 2 runs of `test1`.
In the code in the repository, and also in the packet captures, you can see that each run of `test1`, runs the queries below in order, one by one:
first connection:
```
DROP DATABASE IF EXISTS spi_test WITH (FORCE)
```
```
CREATE DATABASE spi_test
```
second connection the same as first connection.
third connection:
```
DROP DATABASE IF EXISTS spi_test_snapshot WITH (FORCE)
```
```
CREATE DATABASE spi_test_snapshot TEMPLATE spi_test
```
all these only for the first run. As mentioned, there are 2 rounds in each capture.

Here is an image from each capture:

First one. pg 15. Last message is 50ms after the first message.
Second one. pg 14. Last message is 37ms after the first message.

All while the environment and all being the same. Same image arch, same code same everything. Official alpine docker images. Run on a M1 MacBook. I've even confirmed the messages in both captures are as identical as possible at Postgres level (e.g. obviously the md5 hashes of the authentication don't match).

The difference in the time comes from the `CREATE DATABASE spi_test` query. It appears that query takes 15ms in postgres 14 and 28ms in postgres 15 from the query-send till the response-send messages, which means almost twice as slow. The slowness comes after the library is done sending the query, which means the library cannot be at fault (taking all I explained above into account).

This issue is not necessarily a big deal to us, but can anyone please explain the why? Or a fix/workaround?
Maybe some kind of default setting changes between pg 14 and 15 that results in this performance regression?!

Attachment Content-Type Size
pg-14-test1-2-rounds-after-fix.pcapng application/octet-stream 21.0 KB
pg-15-test1-2-rounds-after-fix.pcapng application/octet-stream 21.0 KB

Browse pgsql-performance by date

  From Date Subject
Next Message Andrei Lepikhov 2025-05-18 16:54:24 Re: inefficient/wrong plan cache mode selection for queries with partitioned tables (postgresql 17)
Previous Message James Pang 2025-05-15 13:43:17 Re: a lot of session wait on lock relation