Re: Key management with tests

Greetings,

* Bruce Momjian (bruce(at)momjian(dot)us) wrote:
> On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote:
> > > I looked further. First, I don't think we are going to be able to
> > > protect at all against users who have _write_ access on the OS running
> > > Postgres. It would be too easy to just read process memory, or modify
> > > ~/.profile.
> >
> > I don't think anyone is really expecting that we'll be able to come up
> > with a way to protect against attackers who have fully compromised the
> > OS to the point where they can read/write OS memory, or even the PG unix
> > account. I'm certainly not suggesting that there is a way to do that or
> > that it's an attack vector we are trying to address here.
>
> OK, that's good.
>
> > > I think the only possible option would be to try to give some protection
> > > against users with write access to PGDATA, where PGDATA is on another
> > > server, e.g., via NFS. We can't protect against all db modifications,
> > > for reasons outlined above, but we might be able to protect against
> > > write users being able to _read_ the keys and therefore decrypt data.
> >
> > That certainly seems like a worthy goal. I also really want to stress
> > that I don't think anyone is expecting us to be able to "protect"
> > against users who have write access to the system- write access to files
> > is really an OS level issue and there's not much we can do once someone
> > has found a way to circumvent that (we can try to help the OS by doing
> > things like using SELinux, of course, but that's a different
> > discussion). At the point that an attacker has gotten write access, the
>
> Agreed.
>
> > best we can do is complain loudly if we detect unexpected modifications.
> > Ideally, we would be able to do that for everything, but certainly doing
> > it for the principal data would go a long way and is far better than
> > nothing.
>
> I disagree. If we only warn about some parts, attackers will just
> attack other parts. It will also give users a false sense of security.
> If you can get the keys, it doesn't matter if there is one or ten ways
> of getting them, if they are all of equal difficulty. Same with
> modifying the system files.

I agree that there's an additional concern around the keys and that we
would want to have a solid way to avoid having them be compromised. We
might not be able to guarantee that attackers who can write to PGDATA
can't gain access to the keys in the first implementation, but I don't
see that as a problem- the TDE capability would still provide protection
against improper disposal and some other use-cases, which is useful. I
do think it'd be useful to consider how we could provide protection
against an attacker who has write access from being able to acquire the
keys, but that seems like a tractable problem. Following that, we could
look at how to provide integrity checking for principal data, using one
of the outlined approaches or maybe something else entirely. Lastly,
perhaps we can find a way to provide confidentiality and integrity for
other parts of the system.

Each of these steps is a useful improvement in its own right and will
open up more opportunities for PG to be used. It wasn't my intent to
suggest otherwise, but rather to see if there was an opportunity to get
a few things done at once if it wasn't too impactful. I agree now that
it makes sense to focus on the first step, so we can hopefully get that
accomplished.

> > There are certainly already users out there who intentionally make
> > postgresql.auto.conf owned by root/root, zero-sized, and monitor it to
> > make sure that it isn't updated. postgresql.conf actually is also often
> > monitored for changes by a change management system of some kind and may
> > also be owned by root/root already. I suspect that postmaster.opts is
> > not monitored as closely, but that's probably due more to the fact that
> > we don't really document it as a configuration system file and it can't
> > be put outside of PGDATA. Having a way to move it outside of PGDATA or
> > just not have it be used at all (do we really need it..?) would be
> > another way to address that risk though.
>
> I think postmaster.opts is used for pg_ctl reload. I think the question
> is whether the value of maliciously writable PGDATA being able to read
> the keys, while not protecting or detecting all malicious
> writes/db-modifications, is worth it. And, while I listed the files
> above, there are probably many more ways to break the system.

postmaster.opts is used for pg_ctl restart, just to be clear.

As I try to state above- I don't think we need to provide any specific
protections against a malicious writer for plain encryption to be
useful for some important use-cases. Providing protections against a
malicious writer being able to access the keys is certainly important
as, if they acquire the keys, they would be able to trivially both
decrypt the data and modify any other data they wished to, so it seems
likely that solving that would be the first step towards protecting
against a malicious writer, after which it's useful to think about what
else we could provide integrity checking of, and principal data strikes
me as the next sensible step, followed by what's essentially metadata.

> > > The problem is that this is a limited use-case, and there are probably
> > > other problems I am not considering. It seems too error-prone to even
> > > try protect against this, but it does limit the value of this feature.
> >
> > I don't think we need to consider it a failing of the capability every
> > time we think of something else that really should be addressed when
> > considering this attack vector. We aren't going to be releasing this
> > and saying "we guarantee that this protects against an attacker who has
> > write access to PGDATA". Instead, we would be documenting "XYZ, when
> > enabled, is used to validate the integrity of ABC data. Individuals
> > concerned with unexpected modifications to their system should consider
> > independently monitoring files D, E, F. Note that there is currently no
> > explicit protection against or detection of unexpected or malicious
> > modification of other parts of the system such as the transaction
> > record.", or something along those lines. Hardening guidelines would
> > also recommend things like having postgresql.conf moved out of PGDATA
> > and owned by root/root, etc. Users would then have the ability to
> > evaluate if what we're providing is sufficient for their requirements
> > or not, and to then provide us with feedback about what they feel is
> > still missing before they would be able to use PG for their use-case.
>
> See above --- I think we can't just say we close _most_ of the doors
> here, and I am afraid there will be more and more cases we miss. It
> feels too open-ended. For example, imagine modifying a PGDATA file so
> it is a symbolic link to another file that is not in PGDATA? Seems that
> would break all sorts of security restrictions, and that's just a new
> idea I came up with today.

It's not clear how that would provide the attacker with much, if
anything.

> What I don't want to do is to add a lot of complexity to the system, and
> not really gain any meaningful security.

Integrity is very meaningful to security, but key management would
certainly come first because if an attacker is able to acquire the keys
then they can circumvent any integrity check being done by simply using
the key. I appreciate that protecting the keys is non-trivial but it's
absolutely critical as everything else falls apart if the key is
compromised. I don't think we should be thinking that we're going to be
done with key management or with providing ways to acquire keys even if
the currently proposed patches go in- we'll undoubtably need to provide
other options in the future. There's an interesting point in this
regarding how the flexibility of the shell-script based approach also
introduces this risk that an attacker could modify it and write the key
out to somewhere that they could get at pretty easily. Having support
for directly fetching the key from the Linux kernel or the various
vaulting systems would avoid this risk, I would think. Maybe there's a
way to get PG to dump the key out of system memory by modifying other
files in PGDATA but that's surely quite a bit more difficult.
Ultimately, I don't think this voids the proposed approach but I do
think it means we'll want to improve on this in the future.

> > To that end, I would hope that we'd eventually develop a way to detect
> > unexpected modifications in other parts of the system, both as a way to
> > discover filesystem corruption earlier but also in the case of a
> > malicious attacker. The latter would involve more work, of course, but
> > it doesn't seem insurmountable. I don't think it's necessary to get
> > into that today though.
> >
> > I am concerned when statements are made that we are just never going to
> > do something-or-other because we think it'd be a lot of source code
> > changes or won't be completely perfect against every attack we can think
> > of. There was a good bit of that with RLS which also made it a
> > particularly difficult feature to push forward, but, thanks to clearly
> > documenting what was and wasn't addressed, clearly admitting that there
> > are covert channel attacks that might be possible due to how it works,
> > it's been pretty well accepted and there hasn't been some huge number of
> > issues or CVEs that have been associated with it or mismatched
> > expectations that users of it have had regarding what it does and
> > doesn't protect against.
>
> Oh, that is a very meaningful lesson. I do think that for cluster file
> encryption, if we have a vulnerability, someone will write a script for
> it, and it could be widely exploited. I think RLS gets a little more
> flexibility since someone is already in the database when using it.

In the current attack we're contemplating, the attacker's got write
access to the filesystem and if that's happening then they've managed to
get through a few layers already, I would think, so it seems unlikely
that it would be widely exploited. Of course, we'd like to avoid having
vulnerabilities where we can, but a particular behavior is only a
vulnerabiliy if there's an expectation that we protect against that kind
of attack, which is why documentation is extremely important, which is
what I was trying to get at with the RLS example.

> I am not against adding more security features, but I need agreement
> that the existing features/protections, with the planned source code
> impact, is acceptable. I don't want to go down the road of getting the
> feature with the _hope_ that later changes will make the feature
> acceptable --- for me, either what we are planning now is acceptable
> given its code impact, or it is not. If the feature is not sufficient,
> then I would not move forward until we had a reasonable plan of when the
> feature would have acceptable usefulness, and acceptable source code
> impact.

See above. I do think that the proposed approach is a valuable
capability and improvement in its own right. It seems likely that this
first step, as proposed, would allow us to support use-cases such as the
PCI one you mentioned previously. Taking it further and adding
integrity validation would move us into even more use-cases as it would
address NIST requirements which explicitly call for confidentiality and
integrity.

> The big problem, as you outlined above, is that adding to the
> protections, like malicious write detection for a remote PGDATA, greatly
> increases the code impact, and ultimately, might be unsolvable.

I don't think we really know that it increases the code impact hugely or
is unsolveable, but ultimately those are really debates for another day
at this point.

Thanks,

Stephen