| From: | Tomás Senart <ts(at)perfloop(dot)ai> |
|---|---|
| To: | pgsql-hackers(at)lists(dot)postgresql(dot)org |
| Cc: | Thomas Munro <thomas(dot)munro(at)gmail(dot)com> |
| Subject: | [PATCH v1] Free oversized decoded WAL records in XLogReaderFree() |
| Date: | 2026-11-02 23:15:13 |
| Message-ID: | CAEEBa3Uai8TGPd6P5SX_fzbu3KHMwSvu9WuqA8JPSPQ1RJR8fQ@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
XLogReaderFree() does not free decoded records that were too large for
the decode buffer. XLogReadRecordAlloc() allocates those separately,
and only XLogReleasePreviousRecord() and ResetDecoder() free them. The
record last returned by XLogReadRecord() stays in the decode queue until
the next read, so if it is larger than the decode buffer (64kB by
default), it leaks when the reader is freed.
Most callers free that memory soon after anyway. The checkpointer does
not. CheckPointTwoPhase() calls XlogReadTwoPhaseData() for each
prepared transaction whose PREPARE record is before the redo pointer,
and that function creates and frees a reader in checkpointer_context,
which is reset only after an error. So each large PREPARE record read
there stays allocated until the checkpointer exits.
The attached SQL file reproduces it. It prepares five transactions
with 20000 subtransactions each (80220-byte PREPARE records), runs
CHECKPOINT after each one, and logs the checkpointer's memory contexts.
"Grand total ... used" on master 8628df8891a:
step master patched
1 331136 248392
2 413880 248392
3 496624 248392
4 579368 248392
5 662112 248392
That is 82744 bytes per transaction without the patch. 15.19 shows the
same growth.
The patch calls ResetDecoder() in XLogReaderFree() before the decode
buffer is freed. ResetDecoder() frees only the separately allocated
records, so a decode buffer supplied by the caller is not touched. The
leak goes back to 3f1ce97346 in 15, so I think it should be
backpatched.
With assertions enabled, the regress, recovery, pg_waldump, pg_rewind,
pg_walinspect, test_decoding and pg_walsummary suites give the same
results with and without the patch (the injection-point tests were
skipped).
The leak was found and fixed by Perfloop. I am the contact.
Feel free to rework the patch however you prefer.
Thanks,
Tomás
| Attachment | Content-Type | Size |
|---|---|---|
| v1-0001-Free-oversized-decoded-WAL-records-in-XLogReaderF.patch | text/x-patch | 2.4 KB |
| xlogreader-leak-repro.sql | application/sql | 1.7 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Masahiko Sawada | 2026-11-02 23:41:30 | Re: Parallel vacuum: workers together with leader could overrun cost budget |
| Previous Message | surya poondla | 2026-11-02 23:03:59 | Re: Fix trigger memory leak |