Re: Fix detection of truncated zstd-compressed backups

> On Aug 10, 2026, at 14:45, Chao Li <li(dot)evan(dot)chao(at)gmail(dot)com> wrote:
>
>
>
>> On Aug 8, 2026, at 08:00, Chao Li <li(dot)evan(dot)chao(at)gmail(dot)com> wrote:
>>
>>
>>
>>> On Aug 8, 2026, at 03:42, Zsolt Parragi <zsolt(dot)parragi(at)percona(dot)com> wrote:
>>>
>>> Hello!
>>>
>>> Isn't this a generic problem, also present in gzip/lz4 compressed backups?
>>
>> Maybe, but I haven’t checked them yet. I will do that next week.
>
> I just verified gzip has the same problem using a similar script:
> ```
> workdir=$(mktemp -d /tmp/gzip-trunc.XXXXXX)
> mkdir "$workdir/truncated"
> dd if=/dev/zero of="$workdir/base.tar" bs=1024 count=2
> gzip -c "$workdir/base.tar" > "$workdir/base.tar.gz"
> size=$(stat -f %z "$workdir/base.tar.gz")
> dd if="$workdir/base.tar.gz" of="$workdir/truncated/base.tar.gz" bs=1 count=$((size - 1))
>
> manifest_prefix=$'{"PostgreSQL-Backup-Manifest-Version": 1,\n "Files": [],\n "WAL-Ranges": [],\n'
> printf '%s' "$manifest_prefix" > "$workdir/manifest-prefix"
> manifest_checksum=$(shasum -a 256 "$workdir/manifest-prefix" | awk '{print $1}')
> printf '%s"Manifest-Checksum": "%s"}\n' "$manifest_prefix" "$manifest_checksum" > "$workdir/truncated/backup_manifest"
>
> gzip -t "$workdir/truncated/base.tar.gz"
> pg_verifybackup -F t -s "$workdir/truncated"
> ```
>
> gzip fails, while pg_verifybackup succeeds.
>
> Looking at the code, astreamer_gzip_decompressor_content() stops processing when it has consumed the input buffer. However, the inflate() doc, see [2] and [3], says that only Z_STREAM_END indicates that decompression has completed. My debugging also shows that, when reading the truncated gzip file, inflate() returns Z_OK, while it returns Z_STREAM_END for a valid gzip file. Therefore, this patch records whether inflate() has returned Z_STREAM_END and checks that state in astreamer_gzip_decompressor_finalize().
>
> Unlike zstd, we don't need to call inflate() again in astreamer_gzip_decompressor_finalize(). Once all input has been consumed without Z_STREAM_END, calling inflate() with no input cannot complete the stream.
>
> See attached 0002 for the fix of gzip streamer. I will check the lz4 streamer next.
>
> Best regards,
> --
> Chao Li (Evan)
> HighGo Software Co., Ltd.
> https://www.highgo.com/
>
>
>
>
> <v2-0001-Fix-detection-of-truncated-zstd-compressed-backup.patch><v2-0002-Fix-detection-of-truncated-gzip-compressed-backup.patch>

Sorry, forgot to add the reference links:

[2] https://zlib.net/zlib_how.html
[3] https://zlib.net/manual.html

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/