Fix detection of truncated zstd-compressed backups

From: Chao Li <li(dot)evan(dot)chao(at)gmail(dot)com>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Fix detection of truncated zstd-compressed backups
Date: 2026-08-07 09:42:53
Message-ID: 5962B878-C43D-4EBC-9E95-1F945CE5E586@gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

One of our users reported this issue to me on PG18. pg_verifybackup reported success for a zstd-compressed backup, but zstdfailed to decompress it. After debugging with the user’s data, I found that the backup was truncated, but pg_verifybackup does not verify that the final zstd frame completed. I also tested current master, where the problem still exists.

I created a simple repro. First, use the following commands to create a fake backup folder and truncate one byte from the compressed tar file. The final zstd command will fail:
```
workdir=$(mktemp -d /tmp/zstd-trunc.XXXXXX)
mkdir "$workdir/truncated"
dd if=/dev/zero of="$workdir/base.tar" bs=1024 count=2
zstd -q -f "$workdir/base.tar" -o "$workdir/base.tar.zst"
size=$(stat -f %z "$workdir/base.tar.zst")
dd if="$workdir/base.tar.zst" of="$workdir/truncated/base.tar.zst" bs=1 count=$((size - 1))

manifest_prefix=$'{"PostgreSQL-Backup-Manifest-Version": 1,\n "Files": [],\n "WAL-Ranges": [],\n'
printf '%s' "$manifest_prefix" > "$workdir/manifest-prefix"
manifest_checksum=$(shasum -a 256 "$workdir/manifest-prefix" | awk '{print $1}')
printf '%s"Manifest-Checksum": "%s"}\n' "$manifest_prefix" "$manifest_checksum" > $workdir/truncated/backup_manifest

zstd -t $workdir/truncated/base.tar.zst
```

However, pg_verifybackup doesn't report an error for the truncated backup:
```
% pg_verifybackup -n -F t -s /tmp/zstd-trunc.YHQ9P4/truncated
backup successfully verified
```

Looking at this code in astreamer_zstd_decompressor_content()
```
ret = ZSTD_decompressStream(mystreamer->dctx,
&mystreamer->zstd_outBuf, &inBuf);

if (ZSTD_isError(ret))
pg_fatal("could not decompress data: %s",
ZSTD_getErrorName(ret));
```

While decompressing the truncated tar file, ZSTD_decompressStream() returns 1. The doc for ZSTD_decompressStream() at [1] says that a return > 0 means that there is still decoding or flushing to do before the current frame is complete. In this case, it may indicate an incomplete frame.

So, this patch records the return value in astreamer_zstd_frame and checks it in astreamer_zstd_decompressor_finalize(). See the attached patch for details.

I also added a loop to call ZSTD_decompressStream() again in astreamer_zstd_decompressor_finalize(), because the doc [1] says that, when output.pos == output.size and the return value is greater than 0, the caller must call ZSTD_decompressStream() again to flush the remaining buffered output.

With the fix, now pg_verifybackup reports a failure:
```
% pg_verifybackup -n -F t -s /tmp/zstd-trunc.YHQ9P4/truncated
pg_verifybackup: error: could not decompress data: compressed stream is incomplete
```

BTW, I would also like to bump my previous patch [2]. After changing fe_utils/astreamer_zstd.c, pg_verifybackup is not rebuilt automatically, which is inconvenient. I felt the pain again while working on this patch.

[1] https://github.com/facebook/zstd/blob/v1.5.7/lib/zstd.h#L887-L905
[2] https://www.postgresql.org/message-id/3B062561-C39C-4367-AB1E-F4C27BC6F736%40gmail.com

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

Attachment Content-Type Size
v1-0001-Fix-detection-of-truncated-zstd-compressed-backup.patch application/octet-stream 3.1 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Nikhil Sontakke 2026-08-07 09:52:03 Hot standby accepts connections (giving wrong results) before consistency after crash reset
Previous Message Daniel Gustafsson 2026-08-07 09:42:15 Re: Random 038_walsnd_shutdown_timeout.pl failure on goldfish