pgsql: Process sync requests incrementally in AbsorbSyncRequests

From: Alexander Korotkov <akorotkov(at)postgresql(dot)org>
To: pgsql-committers(at)lists(dot)postgresql(dot)org
Subject: pgsql: Process sync requests incrementally in AbsorbSyncRequests
Date: 2025-07-27 12:23:12
Message-ID: E1ug0PH-000yp4-30@gemulon.postgresql.org
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-committers

Process sync requests incrementally in AbsorbSyncRequests

If the number of sync requests is big enough, the palloc() call in
AbsorbSyncRequests() will attempt to allocate more than 1 GB of memory,
resulting in failure. This can lead to an infinite loop in the checkpointer
process, as it repeatedly fails to absorb the pending requests.

This commit introduces the following changes to cope with this problem:
1. Turn pending checkpointer requests array in shared memory into a bounded
ring buffer.
2. Limit maximum ring buffer size to 10M items.
3. Make AbsorbSyncRequests() process requests incrementally in 10K batches.

Even #2 makes the whole queue size fit the maximum palloc() size of 1 GB.
of continuous lock holding.

This commit is for master only. Simpler fix, which just limits a request
queue size to 10M, will be backpatched.

Reported-by: Ekaterina Sokolova <e(dot)sokolova(at)postgrespro(dot)ru>
Discussion: https://postgr.es/m/db4534f83a22a29ab5ee2566ad86ca92%40postgrespro.ru
Author: Maxim Orlov <orlovmg(at)gmail(dot)com>
Co-authored-by: Xuneng Zhou <xunengzhou(at)gmail(dot)com>
Reviewed-by: Andres Freund <andres(at)anarazel(dot)de>
Reviewed-by: Heikki Linnakangas <hlinnaka(at)iki(dot)fi>
Reviewed-by: Alexander Korotkov <aekorotkov(at)gmail(dot)com>

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/258bf0a2ea8ff86257f750018bfd44397ce7e554

Modified Files
--------------
src/backend/postmaster/checkpointer.c | 155 +++++++++++++++++++++++++---------
1 file changed, 114 insertions(+), 41 deletions(-)

Browse pgsql-committers by date

  From Date Subject
Next Message Alexander Korotkov 2025-07-27 12:33:03 pgsql: Limit checkpointer requests queue size
Previous Message Michael Paquier 2025-07-27 08:49:36 pgsql: Add assertions for all the required index AM callbacks