| From: | Sehrope Sarkuni <sehrope(at)jackdb(dot)com> |
|---|---|
| To: | Pg Hackers <pgsql-hackers(at)postgresql(dot)org> |
| Subject: | [PATCH] Speed up pg_waldump TAP test and fix some GitHub CI Windows flakiness |
| Date: | 2026-08-29 03:43:54 |
| Message-ID: | CAH7T-araSEdsNpxiKaMOW8kr_ZHhekCLHx5yzm2ksq4FdgULbA@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi hackers,
Was testing a rebased Windows patch using GitHub CI and had some flaky slow
test timeout errors for an unrelated pg_waldump test. On Windows (and only
Windows) it was running for 500+ seconds and timing out. After a bit of
research, I figured it's because of the notoriously slow spoofed IPC that
perl does for the forked pg_waldump invocations (that then gets grepped by
perl).
001_basic.pl dumps the full WAL range, segments and records, a multiple
times per scenario across multiple scenarios, and splits the ~9 MB of
output in Perl each time.
The attached decodes the full range once against the plain data directory,
so every rmgr's desc output is still processed at least once. The
fall-off-the-end checks start at the contrecord LSN, a few records from the
end. Everything else gets --limit, which counts after filtering, so the
--rmgr, --fullpage, --fork, --relation and --block runs stop at the first
few matches. This cuts down the IPC considerably.
While it was written to speed up Windows, it's faster on Linux too. There's
not much to save though as it was already only a couple seconds. On a quiet
Linux box of master vs this patch, meson test pg_waldump/001_basic, 10 runs
each: 4.43s (4.18-4.65) -> 2.88s (2.49-3.13)
Windows should gain a lot more. I only have a few GitHub CI numbers to
compare against, but I'm seeing 300-500s for master down to ~40-90s for the
patch.
Here's a recent 538s run on master:
https://github.com/postgres/postgres/actions/runs/33207297796/job/98971778299#step:14:262
Here's the patched version (72s):
https://github.com/sehrope/postgres/actions/runs/33228440369/job/99036914098#step:14:199
Note that the bare --stats run has --limit option in front of it (rather
than as the last option) on purpose as the reverse failed on Windows.
That's due to a bug in src/port/getopt_long.c handling for optional
arguments. I'm putting together a separate patch for that.
Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/
| Attachment | Content-Type | Size |
|---|---|---|
| 0001-Speed-up-pg_waldump-TAP-test.patch | application/octet-stream | 6.1 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | shihao zhong | 2026-08-29 04:43:16 | Re: Allow tuple visibility checks without hint-bit, maintenance |
| Previous Message | Tom Lane | 2026-08-29 03:27:24 | Re: remove_useless_joins vs. bug #19560 |