Incremental backup after a timeline switch: measurements ========================================================= Production builds (no assertions), Linux x86_64, data directories on tmpfs. Base for every row below: master 126976a8ff plus Fujii's v1 for #19774 ("Fix incremental backup data loss due to WAL summary merge"), which merges summaries in LSN order. On plain master, with summaries merged in readdir order, case A happened to restore correctly (0 of 3) while case C lost (3 of 3). Cases A a timeline 1 summary that starts exactly at the switch point (Ayush's report); repro-A.sh.txt C a timeline 1 summary that starts before the switch point and ends after it, with a VACUUM truncation after the recovery target (this report); repro-C.sh.txt R a prior backup whose WAL range ends after the switch point (Sai Pranav's report); his t/013_diverged_parent.pl, tests 3-4 Outcome per run: "ok" = restored rows equal the promoted server's rows, "LOSS" = they differ, "refused" = the incremental backup errors out. R: "accepted" = his tests 3-4 fail (the backup succeeds), "refused" = they pass. 1. Each case against each patch (3 runs per cell) patches on top of the base case A case C case R ----------------------------- ------------ ------------ -------- none LOSS 3/3 LOSS 3/3 accepted Ayush's v1 ok 3/3 LOSS 3/3 accepted Sai Pranav's v1 LOSS 3/3 LOSS 3/3 refused Ayush's v1 + Sai Pranav's v1 ok 3/3 LOSS 3/3 refused + 0001 ok 3/3 refused 3/3 refused + 0001 + 0002 ok 3/3 (*) ok 3/3 (*) refused 0001 and 0002 are applied on top of Ayush's and Sai Pranav's v1 patches as posted. (*) With 0002 the precondition of both scripts never appears: no timeline 1 summary goes past the recovery target before promotion (default WAIT=summary gives up waiting for one, 1 of 1). These cells use WAIT=receive, which waits until the standby has received all the WAL and summarization has reached replay. On the base without patches WAIT=receive loses too: A 3/3, C 3/3. Controls, case C with nothing destructive after the target (JUNK=none): restores with 0001 (1/1) and with 0001 + 0002 (1/1). t/014 and t/015 pass on 0001 (11 tests) and on 0001 + 0002 (15 tests). 2. Case C without patches (repro-C.sh.txt, default WAIT=summary) Branches, plain heads of today, 5 runs each: REL_19_STABLE fc189eb7508, REL_18_STABLE 9412c35035a, REL_17_STABLE e05b7e2a45c: LOSS 5/5 each. On the base (master+F): recovery target kinds (KIND=lsn-incl, xid, time, name): LOSS 3/3 each (the default KIND=lsn is the 3/3 of table 1) recovery_target_action = promote (ACTION=promote): ok 5/5; replay reaches the target before a summary past it is written, so it needs a pause at the target, which is the default action other operations after the target (JUNK=drop, truncate, vacuumfull, cluster, dropdb, none): ok 2/2 each; they leave a limit block of 0, which copies the whole file, or only extra block references 3. What 0002 costs: incremental backups taken ON A STANDBY (lag.sh.txt) Full backup taken on the standby; then two batches of changes, each followed by a checkpoint on the primary; the incremental backup is taken on the standby when it has replayed the first checkpoint and: none has also replayed the second batch delay has received it, but recovery_min_apply_delay = 20s holds it pause has received it, with replay paused before it The restored table must be one of the three states of the primary's history. Base = master+F as above, 3 runs per cell, elapsed time of pg_basebackup --incremental (coarse: shared build lane, not a clean benchmark lane). mode base: result elapsed (s) + all four: result elapsed (s) ------ ------------ ------------------- ------------------ ----------- none ok 3/3 0.115 0.135 0.138 ok 3/3 0.136 0.137 0.051 delay ok 3/3 0.084 0.076 0.078 ok 3/3 0.060 0.058 0.064 pause ok 3/3 0.092 0.081 0.078 ok 3/3 0.059 0.070 0.064 ("all four" = Ayush's v1 + Sai Pranav's v1 + 0001 + 0002.) The backup starts at a restartpoint, which is never ahead of replay, so it never needed summaries beyond replay. In these runs the summarizer's position just before the backup was similar in both builds (its polling interval dominates); with 0002 it cannot pass replay by construction, which t/014 and t/015 check. 4. Tests (assertions + injection points, master 126976a8ff, with Ayush's and Sai Pranav's v1 patches underneath) 0001 (with the summarizer fix of the "Assertion failure in WAL summarizer" thread underneath, which t/014 and t/015 need on assertion-enabled builds): make check, pg_walsummary and pg_combinebackup (t/008, t/013, t/014, t/015) pass. With 0001's basebackup_incremental.c change reverted, t/015 fails; t/013 and t/014 keep passing, as they test the patches underneath. 0001 + 0002 (without that summarizer fix): make check, pg_walsummary, pg_combinebackup and src/test/recovery pass, with no assertion failure in the pg_combinebackup logs. With 0002 reverted, t/014 and t/015 fail. 5. Shihao's test, and a summary written before a minor update (25 Oct) Production builds of master 126976a8ff ("master") and of master + Ayush's v1 + Sai Pranav's v1 + 0001 + 0002 ("the series"), without Fujii's #19774 fix underneath (it does not change case C, see 1). Shihao's t/014_truncation_after_target.pl as posted, 3 runs each: master: fails 3/3, test 2 'restored backup has the update' got "4|0", expected "4|1" the series: passes 3/3 (copied as t/016, t/014 is taken) Old summary (repro-C-upgrade.sh.txt, default knobs, 3 runs each): everything up to the promotion runs on master, so the standby writes the timeline 1 summary 0/03000028 .. 0/0303E500 across the switch point 0/03034520 ("limit 4" for t). The promoted server is then restarted and takes the incremental backup. restarted on master (control): DATA LOSS 3/3 restarted on the series: refused 3/3, with ERROR: cannot use WAL summary file "pg_wal/summaries/ 000000010000000003000028000000000303E500.summary" for an incremental backup The same LSNs in all 6 runs. After the restart on the series, with 0002 running, that summary is still listed: 0002 changes where the summarizer stops, not the summaries already on disk. 0002 alone was not built; its only non-test change is in walsummarizer.c, so its backup path is master's and, from reading the code, it would behave like the control.