Hi,
Over the past couple of days I've been doing a bit of benchmarking for
the "group clog" patch [1], and I've ran into what I suspect might be a
fairly serious performance regression on newer kernels (essentially
4.3.0 and newer). I got to a point where I need help with further
investigation, testing on other systems etc.
The workload tested in the patch [1] is quite simple - a transaction
with 3 x SELECT FOR UPDATE queries and 2 x SAVEPOINT on unlogged tables.
The results (average tps from 5 x 5 minute runs, for 32 and 64 clients)
on multiple kernels look like this:
kernel 32 64
---------------------------------
3.19.8 48524 59291
4.1.33 47193 59574
4.2.8 48901 59877
4.3.0 32187 38970
4.3.6 31889 38815
4.4.0 31946 37702
4.4.23 31498 37724
4.5.5 31531 37351
4.7.6 32859 38490
Notice the sudden drop from ~50k to ~30k tps between 4.2.8 and 4.3.0
(for 32 clients) and from 60k to 40k (for 64 clients). See the attached
kernel-regression-e5-4620.png.
Those results are from a 4-socket machine, with e5-4620 CPUs, so 32
physical cores in total, 64 with HT. The CPU is v1 model (Sandy Bridge
EP, releases in 2012 and discontinued in Q2 2015), so not particularly
new or obsolete.
This is on scale 300, which easily fits into RAM on the machine. The
results are very stable and IMHO quite consistent.
I've also done some tests with regular pgbench (both read-only and
read-write), with WAL-logged tables, and the results are quite similar.
type kernel 32 64
--------------------------------------------
ro 3.19.8 55796 81563
4.4.23 38188 50983
--------------------------------------------
rw 3.19.8 32282 46234
4.4.23 23367 31311
I've tried to reproduce the issue on another machine, but without much
success. This machine however only has 2 sockets and much newer CPU
(e5-2620 v4, so Broadwell, released Q1 2016).
So it might be somewhat related to the older CPU, maybe a slightly
different kernel config, or something else.
If you have access to similar machines (2 or 4 sockets), it'd be very
helpful if you could repeat the benchmarks and report the results, to
confirm (or invalidate) my results.
The test scripts (and results), and kernel configs are available here:
https://bitbucket.org/tvondra/kernel-perf-regression
It's nothing fancy, mostly trivial shell scripts (you'll need to modify
some paths in those, I guess). Testing a single kernel version takes
roughly 1h.
Those are vanilla kernels, BTW - no customized distribution kernels with
extra patches, etc.
[1]
https://www.postgresql.org/message-id/flat/CAA4eK1+8=X9mSNeVeHg_NqMsOR-XKsjuqrYzQf=iCsdh3U4EOA(at)mail(dot)gmail(dot)com
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
