| From: | Raja Sai pranav <rajasaipranav0(at)gmail(dot)com> |
|---|---|
| To: | rajeshkumarth(at)google(dot)com |
| Cc: | masao(dot)fujii(at)gmail(dot)com, pgsql-hackers(at)lists(dot)postgresql(dot)org, pgsql-bugs(at)lists(dot)postgresql(dot)org, kehan5800(at)gmail(dot)com, robertmhaas(at)gmail(dot)com, dilipkumarb(at)google(dot)com |
| Subject: | Re: BUG #19776: pg_combinebackup ignores the truncation_block_length of an intermediate incremental backup |
| Date: | 2026-11-18 13:05:02 |
| Message-ID: | CAPHduZCiuwVZYLS_3T7FXaH0eaKr8fup-eB1RS-x3hHsn25g2w@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-bugs pgsql-hackers |
Hi Rajesh,
I tested v5 on master (45047c51aa).
v5-0001's reconstruct.c change is the same as v4. With v5-0001 and v5-0002
applied,
https://www.google.com/url?q=http://011_ib_truncation.pl&source=gmail&ust=1795093501571000&sa=E
passes. With only v5-0002 (master's reconstruct.c), the new subtest fails
with "got 8, expected 7", so the test catches the bug.
Both look good to me. Which to take is Fujii-san's call, but I'd include
0002, since it doesn't depend on COPY batching or shared_buffers.
Thanks,
Sai Pranav
On Tue, Nov 17, 2026 12:24 PM, Rajesh Kumar Thandapani <
rajeshkumarth(at)google(dot)com> wrote:
> Hi Sai Pranav, Fujii-san,
>
> Thanks for testing that, Sai Pranav!
>
> I have attached v5 split into two patches so Fujii-san can easily pick
> either option:
> 1. v5-0001 contains just the reconstruct.c fix (same as v4).
> 2. v5-0002 adds the deterministic zero-block append test to
> 011_ib_truncation.pl (with the primary stopped while appending to the
> file, and a comment explaining why the zero page is appended directly).
>
> I am okay with either patch (with or without the test) though I feel the
> test adds value by catching any issues immediately.
> Thanks.
>
> Regards,
> Rajesh Kumar Thandapani
>
> On Tue, Nov 17, 2026 at 4:17 AM Raja Sai pranav <rajasaipranav0(at)gmail(dot)com>
> wrote:
>
>> Hi Rajesh,
>>
>> I tried the zero-block variant on current master (5ffc821037) with v4's
>> reconstruct.c, adding the append plus a second incremental backup and a
>> third node to https://www.google.com/url?q=http://011_ib_truncation.pl&
>> source=gmail&ust=1794975425465000&sa=E, and ran it with shared_buffers
>> at 8MB and 128MB. With the fix it passes at both sizes; without the fix it
>> fails at both with 8 rows instead of 7. So unlike the v3 subtest it
>> reproduces the bug deterministically, and nothing in it depends on COPY
>> batching.
>>
>> I think it is worth adding. The only thing I would add is a comment
>> saying why a zero page is appended to the heap file directly, since that is
>> the exact shape the bug mishandles (an unwritten block past the truncation
>> point).
>>
>> Regards,
>> Sai Pranav
>>
>> On Mon, Nov 16, 2026 07:06 PM, Rajesh Kumar Thandapani <
>> rajeshkumarth(at)google(dot)com> wrote:
>>
>>> Hi Fujii-san, Sai Pranav,
>>>
>>> Thank you both for checking that. I agree that relying on COPY batching and
>>> shared_buffers makes the test too sensitive to internal heuristics, so I
>>> have removed the test change in the attached v4 patch.
>>>
>>> One alternative I was wondering about: in 011_ib_truncation.pl, after taking
>>> the first incremental backup (where table t is truncated from 8 blocks to
>>> 7 blocks), we could append one block of zeroes directly to the relation file
>>> on disk from Perl before taking a second incremental backup:
>>>
>>> my $relpath =
>>> $primary->safe_psql('postgres', "SELECT pg_relation_filepath('t');");
>>> my $segfile = $primary->data_dir . "/$relpath";
>>> $segfile .= ".1" if -f "$segfile.1";
>>> open(my $fh, '>>:raw', $segfile) or die "open $segfile: $!";
>>> print $fh ("\0" x $block_size);
>>> close($fh);
>>>
>>> That simulates an unwritten zero-filled page at block 7 without relying on
>>> COPY or shared_buffers, and deterministically reproduces the bug (returning
>>> 8 rows instead of 7 without the fix).
>>>
>>> Do you think a test like that is worth adding, or is it better to leave out
>>> the test and go with v4 as-is?
>>> I am okay with either option; I'm just sharing my thoughts on adding a
>>> test without assuming any internal huristics.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Rajesh Kumar
>>>
>>> On Mon, Nov 16, 2026 at 11:33 AM Raja Sai pranav <
>>> rajasaipranav0(at)gmail(dot)com> wrote:
>>>
>>>> Hi Fujii-san,
>>>>
>>>> I checked this by running https://www.google.com/url?q=
>>>> http://011_ib_truncation.pl&source=gmail&ust=1794895380937000&sa=E
>>>> against current master without the reconstruct.c change, varying $copy_rows
>>>> (1, 4, 16, 17, 18, 32, 100) and shared_buffers (8MB, 16MB, 128MB). The new
>>>> subtest catches the bug in only 6 of the 21 combinations (copy_rows 16-17
>>>> with 8MB, 17-18 with 16MB and 128MB); in all the others it passes without
>>>> the fix. So I agree it depends too much on COPY's batching to be a reliable
>>>> regression test, and dropping it makes sense, since the reproducer upthread
>>>> already demonstrates the bug and the fix.
>>>>
>>>> Regards,
>>>> Sai Pranav
>>>>
>>>> On Mon, Nov 16, 2026 10:42 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
>>>> wrote:
>>>>
>>>>> On Sun, Nov 8, 2026 at 12:21 AM Raja Sai pranav
>>>>> <rajasaipranav0(at)gmail(dot)com> wrote:
>>>>> > - v3 now adds the bulk-extension regression test I'd flagged as
>>>>> missing: the
>>>>> > node3 / incr2-incr4 chain that bumps shared_buffers so COPY
>>>>> bulk-extends the
>>>>> > relation past an intermediate truncation point, leaving zero-filled
>>>>> unwritten
>>>>> > pages in the truncated range.
>>>>>
>>>>> I just wonder if the added test might be fragile. It relies on internal
>>>>> logics, such as COPY's batching behavior ($copy_rows = 17), buffer pin
>>>>> limits (shared_buffers = 8MB), and the bulk-extension strategy leaving
>>>>> pages unwritten. Changes to any of these could easily make the test
>>>>> pass
>>>>> even without the fix. No?
>>>>>
>>>>> If so, isn't it better to leave out the test for now? Also because
>>>>> the reproducer has already demonstrated the bug and the fix.
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> --
>>>>> Fujii Masao
>>>>>
>>>>
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Ishjaap Singh | 2026-11-18 17:39:03 | Re: ECPG: NULL pointer dereference in EXECUTE with a NULL parameter |
| Previous Message | Tender Wang | 2026-11-18 12:30:20 | Re: BUG #19912: MERGE NOT MATCHED BY SOURCE sees stale source values after concurrent update (EPQ) |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Hayato Kuroda (Fujitsu) | 2026-11-18 13:11:32 | RE: Hash index: stop registering pin-only bucket pages in WAL (remove REGBUF_NO_CHANGE) |
| Previous Message | Raja Sai pranav | 2026-11-18 13:04:51 | Re: Assertion failure in WAL summarizer |