Add checksums without --initdb

So on #postgresql, I was musing about methods of getting checksums enabled/disabled without requiring a separate initdb step and minimizing the downtime required to get such functionality enabled.

What about adapting pg_basebackup to add the following options:

-k|--checksums - build the replica with checksums enabled.
-K|—no-checksums - build the replica with checksums disabled.

The way this would work would be to have pg_basebackup's ReceiveAndUnpackTarFile() calculate and/or remove the checksums from each heap page as it is streamed and update the pg_control file to reflect the new checksums setting. After this checksum-enabled replica is created, then it could stream/process WAL and get caught up, then the user fails over to their brand-spanking-new checksum-enabled database. Obviously this would be a bit slower to calculate each page’s checksum than it would be just to write the data out from the tar stream, but it seems to me like this is a single point where the whole database would need to be processed page-by-page as it is.

Possible concerns here are whether checksums are included in WAL full_page_writes or if they are independently calculated; if the latter I think we’d be fine. If checksums are all handled at the layer below WAL than any streamed/processed changes should be fine to get us to the point where we could come up as a master.

We’d also need to be careful to add checksums to only heap files, but that would be able to be handled via the filename prefixes (base|global) (I’m not sure if the relation forks are in standard Page format, but if not we could exclude those as well). Obviously this bakes quite a bit of cluster structural awareness into pg_basebackup and may tie it more strongly to a specific major version, but it seems to me like the tradeoffs would be worth it if you wanted to have that option and the code paths could exist to keep the existing behavior if so.

Andres suggested a separate tool that would basically rewrite the existing data directory heap files in place, which I can also see a use case for, but I also think there’s some benefit to be found in having it happen while the replica is being streamed/built.

Ideas/thoughts/reasons this wouldn’t work?

David
--
David Christensen
PostgreSQL Team Manager
End Point Corporation
david(at)endpoint(dot)com
785-727-1171