Re: pg_waldump: support decoding of WAL inside tarfile

From: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Tomas Vondra <tomas(at)vondra(dot)me>, Andres Freund <andres(at)anarazel(dot)de>, Michael Paquier <michael(at)paquier(dot)xyz>, Andrew Dunstan <andrew(at)dunslane(dot)net>, Amul Sul <sulamul(at)gmail(dot)com>, Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, Chao Li <li(dot)evan(dot)chao(at)gmail(dot)com>, Anthonin Bonnefoy <anthonin(dot)bonnefoy(at)datadoghq(dot)com>, Fujii Masao <masao(dot)fujii(at)gmail(dot)com>, Jakub Wartak <jakub(dot)wartak(at)enterprisedb(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: pg_waldump: support decoding of WAL inside tarfile
Date: 2026-04-02 22:41:33
Message-ID: CA+hUKG+A-=BKP69ze5oc4O5RSBZtKk7dGoqYLeOUQSo8xugTyg@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri, Apr 3, 2026 at 7:48 AM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> *This* tar defaults to:
> --format=posix -f- -b20 --quoting-style=escape --rmt-command=/usr/bin/rmt
> --rsh-command=/usr/bin/ssh

From some quick googling, OpenSUSE is the only distro I could find
that does this, but the fine manual says they plan to make pax the
default upstream so eventually it might be everywhere:

https://www.gnu.org/software/tar/manual/html_section/Formats.html

> Interestingly, pg_verifybackup's t/003_corruption.pl test also fails
> with the same issue, so apparently this platform is even more
> aggressive about sparse-ifying files than Thomas' FreeBSD box.

Looks like sparse files and BTRFS might be a red herring then, if it's
simply been told to put a pax header on every file?

I think my system might be defaulting to "restricted pax", meaning
that pax headers are added to individual files only if necessary,
path-too-long etc, magic sparse format triggered, etc.

https://man.freebsd.org/cgi/man.cgi?query=libarchive-formats

> I'm inclined to push the logic for selecting these tar options
> into some common subroutine in Test::Utils, rather than having
> two copies (and maybe more later).

How about using --format=ustar, instead of that sparse control stuff?
That solves the BSD tar + ZFS problem and the OpenSUSE GNU tar default
format problem in one fell swoop, and actually says what we want
explicitly.

Should we have a <note> near the --path/--wal-path documentation to
state explicitly that usta format is required, and that pax is not
[yet] supported, and that --format=ustar might be required on some
systems?

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2026-04-02 22:50:50 Re: pg_waldump: support decoding of WAL inside tarfile
Previous Message David Rowley 2026-04-02 22:20:50 Re: Small and unlikely overflow hazard in bms_next_member()