Use a bulk-write buffer access strategy for TOAST inserts

Hi hackers,

Bulk insert operations use a BAS_BULKWRITE strategy for the main relation, but
TOAST rows are inserted without it. Large amounts of TOAST data can therefore
evict useful shared buffers. This has already been discussed in [1].

This patch series proposes a new design allowing TOAST inserts to use the bulk
write buffer access strategy. The changes are split into four patches to ease
review.

The first two patches prepare for 0003. 0001 moves allocation and release of the
buffer access strategy out of BulkInsertState, so the same strategy can be passed
to multiple states. 0002 implements the medium term fix suggested by Andres in
22655aa2313 by preventing a BulkInsertState from being reused across relations.
Together, they allow the main and TOAST relations to have separate insertion
states while using one buffer ring.

So, the 4 patches are:

0001:

Buffer access strategies are generally owned by the operation that creates them.
BulkInsertState is an exception: it owns both its BAS_BULKWRITE strategy and
relation local insertion state. Those objects have different useful lifetimes:
one bulk operation may need several relation local states while all of them should
share the same buffer ring.

Pass the strategy to GetBulkInsertState(), leaving its allocation and release
to the caller.

0002:

BulkInsertState contains relation local state, including a target buffer pin
and relation extension information. Reusing it for another relation requires
resetting all that state correctly. Commits b1ecb9b3fcf and 22655aa2313 fixed
bugs caused by failing to do so.

Make GetBulkInsertState() take a relation and ensure that the resulting state
is used only with that relation.

For partitioned COPY, keep one state per leaf relation and share the caller
owned buffer strategy between them. This removes cross relation reuse while
retaining a single buffer ring for the COPY operation.

0003:

Give each BulkInsertState a state for its TOAST relation. Both states use the same
strategy provided by the caller while keeping their relation local insertion state
separate.

0004:

When a rewrite first needs to store TOAST data, create a BAS_BULKWRITE strategy
and a BulkInsertState for the destination relation. The TOAST state then shares
that strategy. Continue using the existing BulkWriteState for the main relation.

Note that the read path is not handled in this patch series, because TOAST data
can be read outside the scan that produced it, so it needs its own strategy and
lifetime. I plan to work on it once the write path (this thread) goes in.

[1]: https://postgr.es/m/20220622143841.GS22452%40telsasoft.com

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com