BUG #19743: pgcrypto crypt(): sha-crypt rounds= silently wraps modulo 2^32

The following bug has been logged on the website:

Bug reference: 19743
Logged by: Ke
Email address: kehan5800(at)gmail(dot)com
PostgreSQL version: 18.6
Operating system: ubuntu 22
Description:

px_crypt_shacrypt() (contrib/pgcrypto/crypt-sha.c:189 on master) parses the
rounds= option of a caller-supplied sha256crypt/sha512crypt salt with

int srounds = strtoint(num, &endp, 10);

if (*endp != '$')
ereport(ERROR, ... "could not parse salt options");

strtoint() (src/common/string.c) signals overflow only by setting errno to
ERANGE and returns the value cast to int anyway. This caller never looks at
errno, so srounds is the request after a signed 32-bit truncation, and the
clamp-with-NOTICE block that follows (crypt-sha.c:209-226) operates on that
truncated number. The comment above that block says "We don't do this
silently and print a NOTICE in such a case", but for inputs >= 2^32 the
"exceeds maximum" branch can never fire.

Reproduction (any build with --with-openssl):

CREATE EXTENSION pgcrypto;

-- 2^32 + 1000: computed with 1000 rounds, no NOTICE at all
SELECT crypt('password', '$5$rounds=4294968296$saltsalt$');
crypt
---------------------------------------------------------------------
$5$rounds=1000$saltsalt$azOwbpkvuuBKkE82dQPwTsQE8JyT9Fflpr9aKid3aT9

SELECT crypt('password', '$5$rounds=1000$saltsalt$')
= crypt('password', '$5$rounds=4294968296$saltsalt$') AS same_hash;
same_hash
-----------
t

-- 2^32 - 1: the NOTICE names a number the caller never wrote
SELECT crypt('password', '$5$rounds=4294967295$saltsalt$');
NOTICE: rounds=-1 is below supported value (1000), using 1000 instead

-- control: a value that fits in int is clamped and warned about
SET statement_timeout = '2s';
SELECT crypt('password', '$5$rounds=1000000000$saltsalt$');
NOTICE: rounds=1000000000 exceeds maximum supported value (999999999),
using 999999999 instead

The effective count is read back from the returned hash string, which
records the rounds actually used. Walking the 2^32 boundary on master:

requested NOTICE effective
4294966296 "rounds=-1000 is below supported value" 1000
4294967295 "rounds=-1 is below supported value" 1000
4294967296 "rounds=0 is below supported value" 1000
4294967297 "rounds=1 is below supported value" 1000
4294968295 "rounds=999 is below supported value" 1000
4294968296 (none) 1000
4294968300 (none) 1004
8589936592 (none) 2000

Expected: a rounds= value that does not fit in int is treated as out of
range, i.e. clamped to PX_SHACRYPT_ROUNDS_MAX (999999999) with the existing
"exceeds maximum supported value" NOTICE (or rejected), and the NOTICEs
report the value the caller supplied.

Actual: the value is reduced modulo 2^32; when the remainder lands inside
[1000, 999999999] the hash is computed at that work factor with no
diagnostic at all, so rounds=4294968296 yields a 1000-round hash, the
minimum.

gen_salt('sha256crypt', n) cannot produce such a salt (its argument is
int4), so the path is a hand-written or imported salt string, i.e. the path
used to verify or re-hash a hash produced elsewhere. Drepper's reference
implementation, which the comment says is being kept compatible, parses
rounds with strtoul() and clamps; libxcrypt (Ubuntu 22.04's crypt(3))
refuses this setting string and returns the failure token "*0". Neither
reduces it modulo 2^32.

Suggested fix: check errno around the call, as other strtoint() callers do:

errno = 0;
srounds = strtoint(num, &endp, 10);
if (*endp != '$')
ereport(ERROR, ...);
if (errno == ERANGE)
srounds = (num[0] == '-') ? PX_SHACRYPT_ROUNDS_MIN
: PX_SHACRYPT_ROUNDS_MAX;

(or parse into a wider type) and print the supplied string rather than the
truncated int in the two NOTICEs.

Related: the integer options of pgp_sym_encrypt() (s2k-count, s2k-mode,
compress-level, ...) are parsed with atoi() and wrap the same way; that is
already being discussed under BUG #19714, so it is only mentioned here.