[PATCH] Speed up pg_waldump TAP test and fix some GitHub CI Windows flakiness

From: Sehrope Sarkuni <sehrope(at)jackdb(dot)com>
To: Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: [PATCH] Speed up pg_waldump TAP test and fix some GitHub CI Windows flakiness
Date: 2026-08-29 03:43:54
Message-ID: CAH7T-araSEdsNpxiKaMOW8kr_ZHhekCLHx5yzm2ksq4FdgULbA@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi hackers,

Was testing a rebased Windows patch using GitHub CI and had some flaky slow
test timeout errors for an unrelated pg_waldump test. On Windows (and only
Windows) it was running for 500+ seconds and timing out. After a bit of
research, I figured it's because of the notoriously slow spoofed IPC that
perl does for the forked pg_waldump invocations (that then gets grepped by
perl).

001_basic.pl dumps the full WAL range, segments and records, a multiple
times per scenario across multiple scenarios, and splits the ~9 MB of
output in Perl each time.

The attached decodes the full range once against the plain data directory,
so every rmgr's desc output is still processed at least once. The
fall-off-the-end checks start at the contrecord LSN, a few records from the
end. Everything else gets --limit, which counts after filtering, so the
--rmgr, --fullpage, --fork, --relation and --block runs stop at the first
few matches. This cuts down the IPC considerably.

While it was written to speed up Windows, it's faster on Linux too. There's
not much to save though as it was already only a couple seconds. On a quiet
Linux box of master vs this patch, meson test pg_waldump/001_basic, 10 runs
each: 4.43s (4.18-4.65) -> 2.88s (2.49-3.13)

Windows should gain a lot more. I only have a few GitHub CI numbers to
compare against, but I'm seeing 300-500s for master down to ~40-90s for the
patch.

Here's a recent 538s run on master:
https://github.com/postgres/postgres/actions/runs/33207297796/job/98971778299#step:14:262

Here's the patched version (72s):
https://github.com/sehrope/postgres/actions/runs/33228440369/job/99036914098#step:14:199

Note that the bare --stats run has --limit option in front of it (rather
than as the last option) on purpose as the reverse failed on Windows.
That's due to a bug in src/port/getopt_long.c handling for optional
arguments. I'm putting together a separate patch for that.

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/

Attachment Content-Type Size
0001-Speed-up-pg_waldump-TAP-test.patch application/octet-stream 6.1 KB

Browse pgsql-hackers by date

  From Date Subject
Next Message shihao zhong 2026-08-29 04:43:16 Re: Allow tuple visibility checks without hint-bit, maintenance
Previous Message Tom Lane 2026-08-29 03:27:24 Re: remove_useless_joins vs. bug #19560