[PATCH] Fix use-after-free after failed pg_checksum_init

Hi there, hackers!

pg_checksum_init() is documented to return -1 on failure. For the SHA types it first stores the requested type in the checksum context, then calls pg_cryptohash_create() / pg_cryptohash_init(). If create succeeds and init fails, it frees the SHA context but does not clear context->raw_context.c_sha2.

basebackup and pg_verifybackup check that return value. pg_combinebackup does not: write_backup_label(), create_manifest_writer(), reconstruct, and the copy path in pg_combinebackup.c all ignore a failed pg_checksum_init() and later call pg_checksum_update() or pg_checksum_final() with the dangling pointer.

create_manifest_writer() always uses CHECKSUM_TYPE_SHA256 for the manifest itself, so this is not limited to --manifest-checksums=sha256.
The use-after-free is reachable in a frontend OpenSSL build when EVP_DigestInit_ex() fails after a successful create. The in-core cryptohash implementation does not fail after create, so a non-OpenSSL build does not hit this path.

A related issue in pg_checksum_final(): if pg_cryptohash_final() fails, the function returns -1 without freeing the SHA context, and there is no separate cleanup API.

The attached patch is against master. It NULLs the pointer after free on init failure, always frees the SHA context in pg_checksum_final(), and checks pg_checksum_init() / pg_checksum_final() in pg_combinebackup. I also think it should be back-patched to 17, where pg_combinebackup was added.

We found this with Clang Static Analyzer. No new regression test is included, injecting a post-create cryptohash init failure would need a
mock that we do not have today. There is no performance or user-visible behavior change on the success path.

Regards,
Yuriy Grigoryev
Kanatbek Kanybekov