SSI: A patch for a Serializability violation

Hi,

While looking into the SSI implementation in Postgres, I came across a
serializability violation. The attached TAP test in the patch demonstrates
this behavior: without the fix, a transaction that violates serializability
runs without aborting, whereas with the fix, one of the transactions in the
cycle is correctly aborted.

The problem occurs when a serializable transaction is summarized.
CheckTargetForConflictsIn() is called when a tuple's writer checks if
another transaction has taken predicate locks on it. If the transaction
taking the predicate locks is not summarized, there is no issue. However,
if it is summarized, the check references sxact->finishedBefore, which is
never populated for OldCommittedSxact.

Additionally, when predicate locks are summarized, they are not easily
released if certain long-running transactions keep the writable transaction
count greater than zero. The provided script
(ssi_lock_reclamation_repro.sh) uses pgbench to demonstrate that one can
easily run out of shared memory space for predicate locks in the presence
of these transactions.

To solve both problems, I propose adding a finishedBefore field to
predicate locks. This field will be populated when the transaction is
summarized. It can then be used in CheckTargetForConflictsIn() to determine
if the reader overlaps with the writer, resolving the serializability
violation. Furthermore, this field can be used to free up the predicate
lock when SxactGlobalXmin crosses finishedBefore.

Regards,
Vaijayanti Bharadwaj