pg_rewind and hint bits leads to visibility corruption

Hi there,

We've found a visibility corruption bug that can happen if you use
pg_rewind against a standby that has written hint bits. Attached is a
TAP test that reproduces it reliably.

At a high level, this bug is:
1. We write hint bits on standbys
2. pg_rewind doesn't overwrite hint bits written, only pages that are
written via the diverging WAL on the primary
3. a standby that is rewound can have a divergence in which rows are
visible from the new primary.

Postgres attempts to avoid this kind of issue by not marking pages as
dirty when writing hint bits on a standby. But WAL replay can dirty the
same page and lead to those hint bits reaching disk.

I've reproduced this against main, as well as 18.6.

A high level description of the TAP test follows. There are many
similar scenarios possible; this is one of the simplest/smallest.

1. Setup a streaming replication setup with 3 nodes, A as primary, S1,
S2 as replicas.
2. S2 is our target, S1 is our rewind source
3. On A, create table c with 100 rows and checkpoint.
4. In a background session on A: BEGIN; INSERT INTO c VALUES (999999);,
capture the transaction's xid, then PREPARE TRANSACTION 'p' — a
two-phase
transaction left in limbo. A checkpoint runs on A, and both
standbys replay up to it.
5. Detach S1 from A: clear its primary_conninfo, reload, wait for its
WAL receiver to drop to zero.
6. Kill the background session, rollback the prepared transaction on A,
and let S2 replay the ROLLBACK PREPARED.
7. Stop S2 cleanly.
8. Promote S1, then force a checkpoint on S1
9. Crash-stop A (immediate).
10. On S1, COMMIT PREPARED 'p' — S1 resolves the orphaned prepared
transaction as committed on its own timeline.
11. Run pg_rewind with S1 as source, S2 as target
12. Fix S2's port (pg_rewind copied S1's config), start S2

At this point, we have different results from the following SELECT on
S1 and S2, despite having just successfully done a pg_rewind:

SELECT count(*) FROM c WHERE id = 999999;

Specifically, S1 will return `1` and S2 will return `0`.

Thanks to my coworkers jan(at)planetscale(dot)com and sinjo(at)planetscale(dot)com for
their assistance on this bug.

Tom