track_planning causing performance regression

Hi,

During fully-cached SELECT-only test using pgbench, Postgres v13Beta1 shows
~45% performance drop [2] at high DB connection counts (when compared with v12.3)

Disabling pg_stat_statements.track_planning (which is 'On' by default)
brings the TPS numbers up to v12.3 levels.

The inflection point (in this test-case) is 128 Connections, beyond which the
TPS numbers are consistently low. Looking at the mailing list [1], this issue
didn't surface earlier possibly since the regression is trivial at low connection counts.

It would be great if this could be optimized further, or track_planning
disabled (by default) so as to not trip users upgrading from v12 with pg_stat_statement
enabled (but otherwise not particularly interested in track_planning).

These are some details around the above test:

pgbench: scale - 100 / threads - 16
test-duration - 30s each
server - 96 vCPUs / 768GB - r5.24xl (AWS EC2 instance)
client - 72 vCPUs / 144GB - c5.18xl (AWS EC2 instance) (co-located with the DB server - Same AZ)
v12 - REL_12_STABLE (v12.3)
v13Beta1 - REL_13_STABLE (v13Beta1)
max_connections = 10000
shared_preload_libraries = 'pg_stat_statements'
shared_buffers 128MB

Reference:
1) https://www.postgresql.org/message-id/1554150919882-0.post%40n3.nabble.com

2) Fully-cached-select-only TPS drops >= 128 connections.

Conn v12.3 v13Beta1 v13Beta1 (track_planning=off)
1 6,764 6,734 6,905
2 14,978 14,961 15,316
4 31,641 32,012 36,961
8 71,989 68,848 69,204
16 129,056 131,157 132,773
32 231,910 226,718 253,316
64 381,778 371,782 385,402
128 534,661 ====> 353,944 539,231
256 636,794 ====> 248,825 643,631
512 574,447 ====> 213,033 555,099
768 493,912 ====> 214,801 502,014
1024 484,993 ====> 222,492 490,716
1280 480,571 ====> 223,296 483,843
1536 475,030 ====> 228,137 477,153
1792 472,145 ====> 229,027 474,423
2048 471,385 ====> 228,665 470,238

3) perf - v13Beta1

- 88.38% 0.17% postgres postgres [.] PostgresMain
- 88.21% PostgresMain
- 80.09% exec_simple_query
- 25.34% pg_plan_queries
- 25.28% pg_plan_query
- 25.21% pgss_planner
- 14.36% pgss_store
+ 13.54% s_lock
+ 10.71% standard_planner
+ 18.29% PortalRun
- 15.12% PortalDrop
- 14.73% PortalCleanup
- 13.78% pgss_ExecutorEnd
- 13.72% pgss_store
+ 12.83% s_lock
0.72% standard_ExecutorEnd
+ 6.18% PortalStart
+ 4.86% pg_analyze_and_rewrite
+ 3.52% GetTransactionSnapshot
+ 2.56% pg_parse_query
+ 1.83% finish_xact_command
0.51% start_xact_command
+ 3.93% pq_getbyte
+ 3.40% ReadyForQuery

4) perf - v12.3

v12.3
- 84.32% 0.21% postgres postgres [.] PostgresMain
- 84.11% PostgresMain
- 72.56% exec_simple_query
+ 26.71% PortalRun
- 15.33% pg_plan_queries
- 15.29% pg_plan_query
+ 15.21% standard_planner
+ 7.81% PortalStart
+ 6.76% pg_analyze_and_rewrite
+ 4.37% GetTransactionSnapshot
+ 3.69% pg_parse_query
- 2.96% PortalDrop
- 2.42% PortalCleanup
- 1.35% pgss_ExecutorEnd
- 1.22% pgss_store
0.57% s_lock
0.77% standard_ExecutorEnd
+ 2.16% finish_xact_command
+ 0.78% start_xact_command
+ 0.59% pg_rewrite_query
+ 5.67% pq_getbyte
+ 4.73% ReadyForQuery

-
robins