Re: Trying out libarchive for reading user-generated WAL tarballs

From: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
To: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: Trying out libarchive for reading user-generated WAL tarballs
Date: 2026-04-10 08:14:07
Message-ID: CA+hUKGL5zBUxOZAztyiRK9YMTEabmi_bjL-yuFQ-XO4n2fLWRw@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri, Apr 10, 2026 at 6:33 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> On Sun, Apr 5, 2026 at 2:42 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> > Perhaps that could be fixed if astreamer_file.c provided
> > "astreamer_file_reader" with the same semantics, so that it could
> > unconditionally call astreamer_pull(privateInfo->archive_streamer),
> > instead of doing the read, push-into-stream itself? Just a thought.
>
> That seems to work.

Added to open commitfest as
https://commitfest.postgresql.org/patch/6665/. But I should have
waited for CI before posting the last version, because it eventually
reminded me that Windows needs:

- streamer->file = fopen(pathname, "r");
+ streamer->file = fopen(pathname, "rb");

Attachment Content-Type Size
v3-0001-Provide-astreamer_plain_reader-for-reading-files.patch text/x-patch 6.3 KB
v3-0002-pg_waldump-Read-data-with-astreamer_plain_reader.patch text/x-patch 5.1 KB
v3-0003-libarchive-Add-configure-and-meson-options.patch text/x-patch 9.0 KB
v3-0004-libarchive-Provide-astreamer_libarchive_reader.patch text/x-patch 9.6 KB
v3-0005-fixup-Use-more-efficient-zero-copy-API.patch text/x-patch 4.2 KB
v3-0006-pg_waldump-Use-astreamer_libarchive_reader.patch text/x-patch 3.5 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Lakshmi N 2026-04-10 08:38:53 Add missing CHECK_FOR_INTERRUPTS in autovacuum catalog scan loops
Previous Message Thomas Munro 2026-04-10 07:57:56 Re: pg_waldump: support decoding of WAL inside tarfile