Re: TDE: Benchmarking WAL encryption approaches

> You did share a benchmark result for the normal transactions, but these
> approaches can also affect the recovery time and replication, right? Might be
> better to do as well.

I didn't share them in my previous email because for two reasons:
* I didn't run any read tests before writing that email
* my expectation was that I'll have to do some optimizations before I
get good read results, and I didn't want to delay sharing my initial
benchmarks because of that.

I've been only trying to measure reads for a few days now, so I do not
want to say that I have final proper results, but based on what I see
so far: that assumption was wrong, recovery/replication performance in
quite good.

My test approach is to:
* generate a large amount of WAL without checkpoints
* kill the server
* take a snapshot, measure recovery speed, restore snapshot, repeat

And then I do the same thing with replica catchup time, and I also
tried to measure if I see any difference in replication lag. In all
time measurements, I also logged user/sys times separately. This makes
it clear that in some scenarios the overall time remains the same, but
the user/sys ratio shifts.

I repeated this with three workloads, one using bulk records (e.g.
huge INSERT/COPY commands), another focusing on many small wal records
(small OLTP writes) and a third one inserting mostly full page writes.

All with the usual benchmarking precautions (dropping caches, setting
cpu governor and other settings. without dropping caches I only see
noise).

Replication shows increased user CPU usage, which is more significant
in the perpage version where the current implementantion decrypts on
the primary/encrypts again on the replica. In the perrecord version,
it only has to decrypt the wal records to apply them. But this doesn't
show up in replication performance, I didn't see any increased
replication lag compared to master in any of my tests so far.

Replica catchup shows a ~20% increased user CPU time across all
workloads, and the time impact varies across workloads, but there's no
significant difference between perrecord and perpage. Full page writes
are the worst with ~12% slower catchup time, bulk writes need 5% more
time and for OLTP encryption is invisible.

Recovery shows the same 20% CPU time, just with different overall
additional time numbers: FP 13%, bulk 12%, OLTP same as not encrypted,
only the USR CPU difference is there.