Re: Complete data erasure

On Fri, Apr 10, 2020 at 08:23:32AM +0000, asaba(dot)takanori(at)fujitsu(dot)com wrote:
>Hello,
>
>I was off the point.
>I want to organize the discussion and suggest feature design.
>
>There are two opinions.
>1. COMMIT should not take a long time because errors are more likely to occur.

I don't think it's a matter of commit duration but a question what to do
in response to errors in the data erasure code - which is something we
can't really rule out if we allow custom scripts to perform the erasure.
If the erasure took very long but couldn't possibly fail, it'd be much
easier to handle than fast erasure failing often.

The difficulty of error-handling is why adding new stuff to commit may
be tricky. Which is why I proposed not to do the failure-prone code in
commit itself, but move it to a separate process.

>2. The data area should be released when COMMIT is completed because COMMIT has to be an atomic action.
>

I don't think "commit is atomic" really implies "data should be released
at commit". This is precisely what makes the feature extremely hard to
implement, IMHO.

Why wouldn't it be acceptable to do something like this?

BEGIN;
...
DROP TABLE x ERASE;
...
COMMIT; <-- Don't do data erasure, just add "x" to queue.

-- wait for another process to complete the erasure
SELECT pg_wait_for_erasure();

That means we're not running any custom commands / code during commit,
which should (hopefully) make it easier to handle errors.

>
>These opinions are correct.
>But it is difficult to satisfy them at the same time.
>So I suggest that users have the option to choose.
>DROP TABLE works as following two patterns:
>
>1. Rename data file to "...del" instead of ftruncate(fd,0).
> After that, bgworker scan the directory and run erase_command.
> (erase_command is command set by user like archive_command.
> For example, shred on Linux.)
>
>2. Run erase_command for data file immediately before ftruncate(fd,0).
> Wait until it completes, then reply COMMIT to the client.
> After that, it is the same as normal processing.
>
>If error of erase_command occurs, it issues WARNING and don't request unlink to CheckPointer.
>It’s not a security failure because I think that there is a risk when data area is returned to OS.
>

I think it was already disicussed why doing file renames and other
expensive stuff that could fail is a bad idea. And I'm not sure just
ignoring erase_command failures (because that's what WARNING does) is
really appropriate for this feature.

>I will implement from pattern 2 because it's more similar to user experience than pattern 1.
>This method has been pointed out as follows.
>
>From Stephen
>> We certainly can't run external commands during transaction COMMIT, so
>> this can't be part of a regular DROP TABLE.
>
>I think it means that error of external commands can't be handled.
>If so, it's no problem because I determined behavior after error.
>Are there any other problems?

I'm not sure what you mean by "determined behavior after error"? You
essentially propose to just print a warning and be done with it. But
that means we can simply leave data files with sensitive data on the
disk, which seems ... not great.

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services