Re: refactoring relation extension and BufferAlloc(), faster COPY

From: Andres Freund <andres(at)anarazel(dot)de>
To: Melanie Plageman <melanieplageman(at)gmail(dot)com>, Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>
Cc: Heikki Linnakangas <hlinnaka(at)iki(dot)fi>, Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>, vignesh C <vignesh21(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>, Yura Sokolov <y(dot)sokolov(at)postgrespro(dot)ru>, Robert Haas <robertmhaas(at)gmail(dot)com>
Subject: Re: refactoring relation extension and BufferAlloc(), faster COPY
Date: 2023-03-30 03:02:33
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


Attached is v6. Changes:

- Try to address Melanie and Horiguchi-san's review. I think there's one or
two further things that need to be done

- Avoided inserting newly extended pages into the FSM while holding a buffer
lock. If we need to do so, we now drop the buffer lock and recheck if there
still is space (very very likely). See also [1]. I use the infrastructure
introduced over in that in this patchset.

- Lots of comment and commit message polishing. More needed, particularly for
the latter, but ...

- Added a patch to fix the pre-existing undefined behaviour in localbuf.c that
Melanie pointed out. Plan to commit that soon.

- Added a patch to fix some pre-existing DO_DB() format code issues. Plan to
commit that soon.

I did some benchmarking on "bufmgr: Acquire and clean victim buffer
separately" in isolation. For workloads that do a *lot* of reads, that proves
to be a substantial benefit on its own. For the, obviously unrealistically
extreme, workload of N backends doing
SELECT pg_prewarm('pgbench_accounts', 'buffer');
in a scale 100 database (with a 1281MB pgbench_accounts) and shared_buffers of
128MB, I see > 2x gains at 128, 512 clients. Of course realistic workloads
will have much smaller gains, but it's still good to see.

Looking at the patchset, I am mostly happy with the breakdown into individual
commits. However "bufmgr: Move relation extension handling into
ExtendBufferedRel{By,To,}" is quite large. But I don't quite see how to break
it into smaller pieces without making things awkward (e.g. due to static
functions being unused, or temporarily duplicating the code doing relation


Andres Freund


Attachment Content-Type Size
v6-0001-bufmgr-Fix-undefined-behaviour-with-unrealistical.patch text/x-diff 1.4 KB
v6-0002-Fix-format-code-in-fd.c-debugging-infrastructure.patch text/x-diff 1.3 KB
v6-0003-hio-Release-extension-lock-before-initializing-pa.patch text/x-diff 2.2 KB
v6-0004-hio-Don-t-pin-the-VM-while-holding-buffer-lock-wh.patch text/x-diff 8.0 KB
v6-0005-bufmgr-Add-some-error-checking-around-pinning.patch text/x-diff 3.3 KB
v6-0006-Add-smgrzeroextend-FileZero-FileFallocate.patch text/x-diff 12.0 KB
v6-0007-bufmgr-Add-Pin-UnpinLocalBuffer.patch text/x-diff 6.7 KB
v6-0008-bufmgr-Remove-buffer-write-dirty-tracepoints.patch text/x-diff 3.2 KB
v6-0009-bufmgr-Acquire-and-clean-victim-buffer-separately.patch text/x-diff 28.3 KB
v6-0010-bufmgr-Support-multiple-in-progress-IOs-by-using-.patch text/x-diff 13.3 KB
v6-0011-bufmgr-Move-relation-extension-handling-into-Exte.patch text/x-diff 44.8 KB
v6-0012-Convert-a-few-places-to-ExtendBufferedRel.patch text/x-diff 12.9 KB
v6-0013-heapam-Add-num_pages-to-RelationGetBufferForTuple.patch text/x-diff 6.8 KB
v6-0014-hio-Use-ExtendBufferedRelBy.patch text/x-diff 14.5 KB
v6-0015-WIP-Don-t-initialize-page-in-vm-fsm-_extend-not-n.patch text/x-diff 2.1 KB
v6-0016-Convert-a-few-places-to-ExtendBufferedRelTo.patch text/x-diff 11.6 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Masahiko Sawada 2023-03-30 03:15:29 Re: logical decoding and replication of sequences, take 2
Previous Message Amit Kapila 2023-03-30 03:01:39 Re: logical decoding and replication of sequences, take 2