Re: Key management with tests

I have made significant progress on the cluster file encryption feature so
it is time for me to post a new set of patches.

I went backward and forward on this patch. I went backward and created
significant user and developer documentation for this feature. I went
forward and added heap/index file encryption, so it actually does
something useful.

Patch
-----
I have broken the patch into 13 parts, attached, which can be reviewed
separately. However, to get a running system, you have to apply all the
patches in order. The GitHub patch link has more details, has a combined
patch link, and is updated regularly:

https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Patches

I have 13 questions about the patch's approach, marked with XXX
comments. I hope to get these 13 patches into committable shape, but
not commit them, and then continue on to WAL and temporary file
encryption, and pg_basebackup changes. My hope is to have this full
feature committed together early in the PG 15 development cycle. This
progress would not be possible without critical work by Masahiko Sawada
and Stephen Frost.

Performance
-----------
I have used pgbench to test the performance overhead of this feature.
AES128 shows 2.5% overhead, and AES256 shows 3.5% overhead. My testing
was done on this 16-core, 24MB, single SSD server:

https://momjian.us/main/blogs/pgblog/2012.html#January_20_2012

My CPU supports AES acceleration. I made these configuration changes:

shared_buffers = 6GB
effective_io_concurrency = 256
effective_cache_size = 12GB
checkpoint_completion_target = 0.9
max_wal_size = 3GB
wal_compression = on

I used a scale factor of 1000 (15GB database), with this invocation:

pgbench --no-vacuum --protocol prepared --client 32 --jobs 16 --time 3600

I prewarmed the server via pg_prewarm and ran pgbench for an hour before
using the results from another one-hour run. TPS was around 6.5k. For
the default shared_buffers size of 128MB, I see 7% overhead.

--
Bruce Momjian <bruce(at)momjian(dot)us> https://momjian.us
EDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee