BUG #19457: RE: pgp_sym_encrypt silently accepts non-FIPS ciphers (bf, cast5, 3des) when OpenSSL is in FIPS mod

The following bug has been logged on the website:

Bug reference: 19457
Logged by: Shishir Sharma
Email address: ansh01072001(at)gmail(dot)com
PostgreSQL version: 18.3
Operating system: AlmaLinux 8.10, x86_64
Description:

When PostgreSQL 18.3 is built against OpenSSL operating in FIPS mode, the
pgp_sym_encrypt() (and pgp_pub_encrypt()) functions in the pgcrypto
extension silently succeed when called with non-FIPS-approved cipher
algorithms: bf (Blowfish), cast5 (CAST5), and 3des (Triple-DES).

This is a FIPS compliance gap. Every other non-FIPS algorithm in
pgcrypto is either blocked by OpenSSL (e.g. encrypt(..., 'bf'),
digest(..., 'md5')) or, as of PostgreSQL 18, controlled by the new
pgcrypto.builtin_crypto_enabled GUC (gen_salt(), crypt()). The PGP
code path is the only one left completely unguarded.

The result is that a deployment that has gone to the trouble of
enabling OpenSSL FIPS mode expecting that prohibited algorithms
cannot be used can still encrypt production data with Blowfish,
CAST5, or 3DES via pgp_sym_encrypt(), with no error, no warning, and
no indication that a FIPS violation has occurred.

=== Root Cause ===

The pgp_sym_encrypt() call chain is:

  pgp_sym_encrypt()
    → encrypt_internal()        [pgp-pgsql.c]
      → pgp_set_cipher_algo()   [pgp.c]         -- maps 'cast5' →
PGP_SYM_CAST5
      → pgp_encrypt()           [pgp-encrypt.c]
        → pgp_cfb_create()      [pgp-cfb.c]
          → pgp_load_cipher()   [pgp.c]         -- maps PGP_SYM_CAST5 →
"cast5-ecb"
            → px_find_cipher()  [openssl.c]     -- calls
EVP_get_cipherbyname()

The pgcrypto.builtin_crypto_enabled GUC introduced in PostgreSQL 18
(commits 924d89a and 035f99c) protects only px_crypt() and
px_gen_salt() in px-crypt.c. The PGP encryption path never passes
through px-crypt.c and is therefore not covered by the GUC, regardless
of whether it is set to 'on', 'off', or 'fips'.

=== Steps to Reproduce ===

Prerequisites:
  - PostgreSQL 18 built with OpenSSL
  - OpenSSL configured in FIPS mode (fips_mode() returns true)
  - pgcrypto extension installed
  - FIPS enabled system

-- Step 1: Verify FIPS mode is active
SELECT fips_mode();
-- Expected: t

-- Step 2: Confirm the GUC is set to 'fips' (the recommended FIPS setting)
SET pgcrypto.builtin_crypto_enabled = 'fips';
SHOW pgcrypto.builtin_crypto_enabled;
-- Expected: fips

-- Step 3: Verify the GUC correctly blocks gen_salt/crypt (it does work
there)
SELECT gen_salt('bf');
-- Expected: ERROR: use of non-FIPS validated crypto not allowed...
-- Actual:   ERROR: use of non-FIPS validated crypto not allowed...  ✓

-- Step 4: Verify encrypt() correctly blocks non-FIPS ciphers (OpenSSL
blocks these)
SELECT encrypt('secret'::bytea, 'key12345'::bytea, 'bf');
-- Expected: ERROR
-- Actual:   ERROR: encrypt error: Cipher cannot be initialized  ✓

-- Step 5: THE BUG — pgp_sym_encrypt silently succeeds with non-FIPS ciphers
SELECT pgp_sym_encrypt('secret', 'key', 'cipher-algo=cast5,
compress-algo=0');
-- Expected: ERROR (non-FIPS algorithm should be rejected)
-- Actual:   \xc30d0403... (ciphertext returned silently — FIPS VIOLATION)

SELECT pgp_sym_encrypt('secret', 'key', 'cipher-algo=bf, compress-algo=0');
-- Expected: ERROR
-- Actual:   \xc30d0404... (ciphertext returned silently — FIPS VIOLATION)

SELECT pgp_sym_encrypt('secret', 'key', 'cipher-algo=3des,
compress-algo=0');
-- Expected: ERROR
-- Actual:   \xc30d0402... (ciphertext returned silently — FIPS VIOLATION)

-- Step 6: Confirm FIPS-approved ciphers still work correctly
SELECT pgp_sym_encrypt('secret', 'key', 'cipher-algo=aes256,
compress-algo=0');
-- Actual: \xc30d0409... (correct — AES-256 is FIPS approved)  ✓

=== Proposed Fix ===
I am happy to work on this.

Add a FIPS cipher check in pgp_load_cipher() in contrib/pgcrypto/pgp.c.
This function is the single chokepoint for all PGP cipher operations
(encrypt, decrypt, session key encryption/decryption). A whitelist of
FIPS 140-2/140-3 approved ciphers for PGP use would be:

  PGP_SYM_AES_128, PGP_SYM_AES_192, PGP_SYM_AES_256

All other ciphers (PGP_SYM_BLOWFISH, PGP_SYM_CAST5, PGP_SYM_DES3,
PGP_SYM_TWOFISH, etc.) should raise an error when CheckFIPSMode()
returns true.

The error message should be consistent with the one used for gen_salt/crypt:
  ERROR: use of non-FIPS validated crypto not allowed when OpenSSL is in
FIPS mode

Additionally, the pgcrypto documentation (doc/src/sgml/pgcrypto.sgml)
should be updated to note which cipher-algo values for pgp_sym_encrypt
are not FIPS 140-2/140-3 approved (bf, cast5, 3des) and that they will
fail when OpenSSL FIPS mode is active with the fix applied.

Related upstream work for reference:
  - Commit 924d89a: Add fips_mode() SQL function
  - Commit 035f99c: Add pgcrypto.builtin_crypto_enabled GUC

Thank you for your time reviewing this.