lost lock during toasting allows fk violation

Hi,

A row lock taken by a foreign-key check can be silently discarded if it is
acquired while a concurrent non-key UPDATE of the referenced row is inside
its
TOAST step. The referenced row can then be deleted without waiting and
without
error, leaving an orphaned child row.

Reproduced on master at 5a3b22eb304, on two independent machines. Back
branches
not tested.

Setup
-----

CREATE TABLE l4_parent (id int PRIMARY KEY, payload text);
ALTER TABLE l4_parent ALTER COLUMN payload SET STORAGE EXTERNAL;
CREATE TABLE l4_child (cid serial PRIMARY KEY, pid int REFERENCES
l4_parent(id));
INSERT INTO l4_parent VALUES (1,'small');

Deterministic reproduction
--------------------------

To hold the UPDATE inside the TOAST step, session L takes a ShareLock on the
parent's TOAST relation, which blocks any TOAST insert:

-- L (get the toast relation name first)
SELECT reltoastrelid::regclass FROM pg_class WHERE
oid='l4_parent'::regclass;
BEGIN; REINDEX TABLE pg_toast.pg_toast_NNNNN; -- hold open for now

-- session A: non-key UPDATE with a payload that must be TOASTed.
-- Blocks inside the TOAST step, wait_event = Lock/relation.
BEGIN; UPDATE l4_parent SET payload=repeat('q',9000) WHERE id=1;
-- (leave open, then commit after L releases)

-- session B: FK check against the parent row.
BEGIN; INSERT INTO l4_child(pid) VALUES (1);
-- (leave open)

-- session L commits, session A commits.

-- session C: B's transaction is still open.
BEGIN; DELETE FROM l4_parent WHERE id=1; COMMIT;

-- session B: commits last, after the parent row is already gone.
COMMIT;

finally, from a new session, we can confirm l4_parent has zero rows, but
l4_child has an orphan pointing to parent id 1.

Without reindex
----------------------------
I ran a few trials of the same strategy but with no reindex step, relying
instead on the time it takes to insert a very large TOAST value to hold the
lock. Times here are taken from a sample script run:

* A does the UPDATE with a 64 MB EXTERNAL value (566 ms);
* B's FK check is fired at offsets 0.02s-0.35s into it;
* C then attempts the DELETE with statement_timeout='2500ms' while B is
still
open.

All 12 trials produced the orphan. (An earlier run
of the same harness on different hardware, where the UPDATE took 372 ms,
gave
11 of 12, the twelfth landing after the window had closed.)

So, this can be reproduced in the wild without too much hand-holding.

Jacob