Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Aug 14, 2019 at 04:36:35PM +0200, Antonin Houska wrote:
> I can work on it right away but don't know where to start.

I think the big open question is whether there will be acceptance of an
all-cluster encyption feature. I guess if no one objects, we can move
forward.

> First, I think we should use a code repository to integrate [1] and [2]
> instead of sending diffs back and forth. That would force us to resolve
> conflicts soon and help to avoid duplicate work. The diffs would be created
> only whe we need to post the next patch version to pgsql-hackers for review,
> otherwise the discussions of details can take place elsewhere.

Well, we can do that, or just follow the TODO list and apply items as we
complete them. We have found that doing everything in one big patch is
just too hard to review and get accepted.

> The most difficult problem I see now regarding the collaboration is agreement
> on the key management user interface. The Full-Cluster Encryption feature [1]
> should not add configuration variables or even tools that the next, more
> sophisticated version [2] deprecates immediately. Part of the problem is that

Yes, the all-cluster encryption feature has _no_ SQL-level API to
control it, just a GUC variable that you can use SHOW to see the
encryption mode.

> [2] puts all (key management related) interaction of postgres with the
> environment into an external library. As I pointed out in my response to [2],
> this will not work for frontend applications (e.g. pg_waldump). I think the
> key management UI for [2] needs to be designed first even if PG 13 should
> adopt only [1].

I think there are several directions we can go after all-cluster
encryption, and it does matter because we would want minimal API
breakage. The options are:

1) Allow per-table encyption control to limit encryption overhead,
though all of WAL still needs to be encrypted; we could add a
per-record encyption flag to WAL records to avoid that.

2) Allow user-controlled keys, which are always unlocked, and encrypt
WAL with one key

3) Encrypt only the user-data portion of pages with user-controlled
keys. FREEZE and crash recovery work since only the user data is
encrypted. WAL is not encrypted, except for the user-data portion

I think once we implement all-cluster encryption, there will be little
value to #1 unless we find that page encryption is a big performance
hit, which I think is unlikely based on performance tests so far.

I don't think #2 has much value since the keys have to always be
unlocked to allow freeze and crash recovery.

I don't think #3 is viable since there is too much information leakage,
particularly for indexes because the tid is visible.

Now, if someone says they still want 2 & 3, which has happened many
times, explain how these issues can be reasonable addressed.

I frankly think we will implement all-cluster encryption, and nothing
else. I think the next big encryption feature after that will be
client-side encryption support, which can be done now but is complex;
it needs to be easier.

> At least it should be clear how [2] will retrieve the master key because [1]
> should not do it in a differnt way. (The GUC cluster_passphrase_command
> mentioned in [3] seems viable, although I think [1] uses approach which is
> more convenient if the passphrase should be read from console.) Rotation of
> the master key is another thing that both versions of the feature should do in
> the same way. And of course, the fronend applications need consistent approach
> too.

I don't see the value of an external library for key storage.

> I'm not too happy to start another (potentially long) discussion in this
> already huge thread, but I think the UI stuff belongs to the -hackers list
> rather than to an offline discussion.

I think the big question is whether we do anything, or just decide we
can't agree and stop.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +