Re: [PoC] Non-volatile WAL buffer

On 26/11/2020 21:27, Tomas Vondra wrote:
> Hi,
>
> Here's the "simple patch" that I'm currently experimenting with. It
> essentially replaces open/close/write/fsync with pmem calls
> (map/unmap/memcpy/persist variants), and it's by no means committable.
> But it works well enough for experiments / measurements, etc.
>
> The numbers (5-minute pgbench runs on scale 500) look like this:
>
> master/btt master/dax ntt simple
> -----------------------------------------------------------
> 1 5469 7402 7977 6746
> 16 48222 80869 107025 82343
> 32 73974 158189 214718 158348
> 64 85921 154540 225715 164248
> 96 150602 221159 237008 217253
>
> A chart illustrating these results is attached. The four columns are
> showing unpatched master with WAL on a pmem device, in BTT or DAX modes,
> "ntt" is the patch submitted to this thread, and "simple" is the patch
> I've hacked together.
>
> As expected, the BTT case performs poorly (compared to the rest).
>
> The "master/dax" and "simple" perform about the same. There are some
> differences, but those may be attributed to noise. The NTT patch does
> outperform these cases by ~20-40% in some cases.
>
> The question is why. I recall suggestions this is due to page faults
> when writing data into the WAL, but I did experiment with various
> settings that I think should prevent that (e.g. disabling WAL reuse
> and/or disabling zeroing the segments) but that made no measurable
> difference.

The page faults are only a problem when mmap() is used *without* DAX.

Takashi tried a patch earlier to mmap() WAL segments and insert WAL to
them directly. See 0002-Use-WAL-segments-as-WAL-buffers.patch at
https://www.postgresql.org/message-id/000001d5dff4%24995ed180%24cc1c7480%24%40hco.ntt.co.jp_1.
Could you test that patch too, please? Using your nomenclature, that
patch skips wal_buffers and does:

clients -> wal segments (PMEM DAX)

He got good results with that with DAX, but otherwise it performed
worse. And then we discussed why that might be, and the page fault
hypothesis was brought up.

I think 0002-Use-WAL-segments-as-WAL-buffers.patch is the most promising
approach here. But because it's slower without DAX, we need to keep the
current code for non-DAX systems. Unfortunately it means that we need to
maintain both implementations, selectable with a GUC or some DAX
detection magic. The question then is whether the code complexity is
worth the performance gin on DAX-enabled systems.

Andres was not excited about mmapping the WAL segments because of
performance reasons. I'm not sure how much of his critique applies if we
keep supporting both methods and only use mmap() if so configured.

- Heikki