BUG #19890: pg_receivewal advances past a truncated LZ4 WAL segment

The following bug has been logged on the website:

Bug reference: 19890
Logged by: Wren
Email address: idrissaajalloh(at)gmail(dot)com
PostgreSQL version: 18.6
Operating system: Linux/amd64
Description:

pg_receivewal treats an incomplete LZ4 frame with a completed WAL filename
as a complete segment when decompression has already produced exactly
WalSegSz bytes. On restart, it advances to the following segment, skipping
validation or reacquisition of the damaged segment.

A completed-name LZ4 segment should be accepted only if decompression
produces exactly WalSegSz bytes and reaches the LZ4 frame-completion state.
A frame truncated after its final output bytes but before its footer is
complete should be rejected or skipped.

How to repeat:

Save the following as run.sh in an empty directory. Run it as `sh run.sh
/absolute/path/to/postgresql-source`. The source tree needs the normal build
prerequisites and the LZ4 development library.

#!/bin/sh
set -eu

if test "$#" -ne 1; then
echo "usage: $0 /absolute/path/to/postgresql-source" >&2
exit 2
fi
src=$1
case "$src" in /*) ;; *) echo "source path must be absolute" >&2; exit 2;;
esac
test -x "$src/configure" || { echo "not a PostgreSQL source tree: $src" >&2;
exit 2; }

base=$(pwd)/work.$$
build=$base/build
inst=$base/install
data=$base/data
sock=$base/socket
wal=$base/wal
mkdir -p "$build" "$inst" "$data" "$sock" "$wal"

cd "$build"
"$src/configure" --prefix="$inst" --with-lz4
make -j2
make install

"$inst/bin/initdb" -D "$data" -A trust --no-locale
cat >>"$data/postgresql.conf" <<EOF
listen_addresses = ''
unix_socket_directories = '$sock'
wal_level = replica
max_wal_senders = 5
EOF
"$inst/bin/pg_ctl" -D "$data" -l "$base/server.log" start
trap '"$inst/bin/pg_ctl" -D "$data" stop -m immediate >/dev/null 2>&1 ||
true' EXIT HUP INT TERM

export PGHOST=$sock PGDATABASE=postgres
# Receive two whole segments. Repeated switches are used because the first
# switch can finish the segment in which initdb/startup left the server.
endpos=$("$inst/bin/psql" -AtX -c "SELECT '0/0'::pg_lsn +
((((pg_current_wal_insert_lsn()-'0/0')::bigint/16777216)+2)*16777216)")
(
sleep 1
"$inst/bin/psql" -X -c 'SELECT pg_switch_wal()' >/dev/null
"$inst/bin/psql" -X -c 'CREATE TABLE lz4_frame_probe(a text); INSERT
INTO lz4_frame_probe SELECT repeat(md5(g::text),100) FROM
generate_series(1,20000) g; SELECT pg_switch_wal(); DROP TABLE
lz4_frame_probe; SELECT pg_switch_wal()' >/dev/null
) &
producer=$!
"$inst/bin/pg_receivewal" -D "$wal" -Z lz4 -E "$endpos" -n -v
>"$base/first.log" 2>&1
wait "$producer"

set -- "$wal"/????????????????????????.lz4
test "$#" -ge 2 || { echo "SETUP ERROR: fewer than two completed LZ4
segments"; exit 2; }
highest=
previous=
for f in $(printf '%s\n' "$wal"/????????????????????????.lz4 | sort); do
previous=$highest; highest=$f; done
test -n "$previous" && test -n "$highest"
before=$(wc -c <"$highest")
truncate -s $((before - 1)) "$highest"

# With the lower complete segment retained, rejection has an unambiguous
# oracle: restart at the corrupted segment, not its successor.
set +e
"$inst/bin/pg_receivewal" -D "$wal" -Z lz4 -E 0/1 -n -v >"$base/second.log"
2>&1
rc=$?
set -e
cat "$base/second.log"

highhex=$(basename "$highest" .lz4)
logid=$(printf '%d' "0x$(printf '%s' "$highhex" | cut -c9-16)")
segid=$(printf '%d' "0x$(printf '%s' "$highhex" | cut -c17-24)")
nextno=$((logid * 256 + segid + 1))
ptr_hi=$((nextno / 256))
ptr_lo=$(((nextno % 256) * 16777216))
successor=$(printf '%X/%08X' "$ptr_hi" "$ptr_lo")
if grep -q "starting log streaming at $successor" "$base/second.log"; then
echo "BUG REPRODUCED: truncated $(basename "$highest") was accepted;
rc=$rc"
exit 1
fi
echo "NOT REPRODUCED: truncated frame was not used as a completed segment;
rc=$rc"
exit 0

The affected behavior is reported when the second pg_receivewal invocation
starts at the successor of the truncated highest segment and the script
prints:

BUG REPRODUCED: truncated <WAL filename>.lz4 was accepted; rc=0

The script then exits with status 1. Exit status 2 indicates a setup
failure. A corrected implementation should reject or skip the truncated
frame, avoid starting at its successor, print `NOT REPRODUCED`, and exit
with status 0.

In the recorded PostgreSQL 18.6 run, a completed segment was truncated by
one byte and pg_receivewal produced:

TRUNCATED_FILE=00000001000000000000001A.lz4 BEFORE=140130 AFTER=140129
pg_receivewal: starting log streaming at 0/1B000000 (timeline 1)
pg_receivewal: stopped log streaming at 0/1B000000 (timeline 1)
SECOND_RC=0

Here 0/1B000000 is the boundary following the truncated
00000001000000000000001A.lz4 segment, showing that directory scanning
classified the incomplete frame as complete.

The issue was reproduced on PostgreSQL 18.6 on Linux/amd64 with LZ4 support.
The LZ4 path used to find the streaming start counts decompressed bytes but
does not require LZ4F_decompress() to return 0, the frame-completion signal.
Consequently, truncating the frame footer can leave the decompressed size
equal to WalSegSz and allow the file to be accepted.

This occurs when pg_receivewal restarts from a directory whose highest
completed-name LZ4 WAL file is truncated after producing a full segment but
before frame completion.