Re: Reduce WAL volume for heap tuple hint bits

On Wed, 19 Aug 2026 at 18:08, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
>
>
>
> > On 17 Aug 2026, at 16:53, Matthias van de Meent <boekewurm+postgres(at)gmail(dot)com> wrote:
>
>
> Hi Matthias, Surya, Ewan,
>
> Surya's reply appears to have been threaded under "Compression of bigger
> WAL records", probably because of a stale References header. I am
> bringing that part of the discussion back to the hint-bit thread here.
>
> Thank you for the reviews. I think they expose two separate questions
> that I had mixed together in v1: protection against page damage and
> propagation of hint bits to standbys.
>
> First, I agree with Matthias that the failure mode is broader than a
> write containing some sectors from the old page and some from the new
> page. An interrupted page write may leave bytes that belong to neither
> version. A record containing only tuple offsets and hint bits cannot
> repair such damage.
>
> PostgreSQL already permits a clean page to be dirtied by a hint-only
> change without an FPI when full_page_writes is on but wal_log_hints and
> checksums are both off. That does not make it a good model to extend.
> We should not use an existing hole in the protection as a reason to
> design a better replicated /dev/null.
>
> Surya wrote that the compact record should advance the page LSN so that
> FlushBuffer() flushes the WAL before the page. I deliberately avoided
> that, although the reason was not explained clearly enough in v1.
>
> For a non-critical hint change, writing the page before the WAL record
> is harmless by itself:
>
> * if the page reaches disk first, the hint is already present;
> * if the WAL record reaches disk first, redo can apply the hint;
> * if neither reaches disk, the hint is lost and can be recomputed.

It actually isn't harmless, because this could cause a cluster to
write ALL_VISIBLE hint bits to disk, for tuples that were modified by
a transaction whose COMMIT record isn't yet replayed when we reach the
end of recovery after a crash. The "write pages only after WAL is
durable" ordering provides protection against this, because it gives a
guarantee that every dirty page on disk is fully backed by WAL.

> PostgreSQL already allows the first case when it sets hints without WAL.
> Thus, an on-disk hint without a corresponding durable WAL record is not
> by itself a violation of the WAL rule for critical data.

I think it is a violation once we introduce durability with WAL. The
assumed guarantee of WAL is that you get a durable prefix of changes,
even in the face of crashes at inopportune moments. Writing data
durably before the WAL is durable is in violation of this. The only
violation of this would currently be async commits, but even that
currently has a consistency guarantee with FPI hint bit logging.
Getting rid of accurate page LSNs would re-introduce phantom
visibility bits.

> Advancing the page LSN would create a different problem. A later
> ordinary modification could see an LSN newer than the checkpoint redo
> pointer and conclude that the page was already protected by an FPI.
> The compact hint record is not such protection, so that could leave the
> ordinary modification exposed to page damage. I do not think we can
> use PageSetLSN() for this record without separating the WAL-before-data
> LSN from the LSN used to decide whether an FPI is required.

There are bits available, both on the page and in buffer headers,
which could be used to store a signal "no, I actually still need an
FPI in this checkpoint window, even if my LSN is from within the
checkpoint window". At the cost of increased complexity in
XLogInsert() and PageSetLsn() handling, of course.

> This suggests reclassifying the proposal. Instead of reducing WAL by
> replacing an FPI, it could deliver heap visibility hints with
> wal_log_hints on, full_page_writes off, and checksums disabled.
> In that configuration the compact record will usually be larger than
> today's block-reference-only FPI_FOR_HINT record, so WAL reduction is no
> longer present. The benefit would be that the standby receives and
> can persist the hints selected by the primary.
>
> Matthias, Surya, Ewan, do you think that is a useful and sound scope for
> the patch? It seems very narrow, so we can just rejection the patch.

I think the scope is very narrow indeed, and I'm not convinced we can
commit to such a patch without losing some consistency/correctness
guarantees that we do get with that specific settings combination.

Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)