BUG #19790: "cannot start a subtransaction ... inval messages" after "ALTER DOMAIN ... DROP CONSTRAINT"

The following bug has been logged on the website:

Bug reference: 19790
Logged by: Alperen Keles
Email address: alpkeles99(at)gmail(dot)com
PostgreSQL version: 18.6
Operating system: macOS 26 arm64
Description:

Hello,

After ALTER DOMAIN ... DROP CONSTRAINT inside a transaction, the next
subtransaction that changes the catalogs fails with an internal error:

CREATE DOMAIN d AS int CHECK (VALUE > 0);
BEGIN;
ALTER DOMAIN d DROP CONSTRAINT d_check;
SAVEPOINT s;
CREATE TABLE x (a int);
ERROR: cannot start a subtransaction when there are unprocessed inval
messages

The same happens with a PL/pgSQL block with an EXCEPTION clause. Here
the ADD CONSTRAINT should fail validation with check_violation and be
caught; instead the XX000 error escapes the handler and aborts the
transaction:

CREATE DOMAIN d2 AS int2 CONSTRAINT d2_k CHECK (VALUE IS NOT NULL);
CREATE TABLE t1 (c0 d2);
INSERT INTO t1 VALUES (-512);
BEGIN;
ALTER DOMAIN d2 DROP CONSTRAINT d2_k;
DO $$BEGIN
ALTER DOMAIN d2 ADD CONSTRAINT d2_k CHECK (VALUE >= -303);
EXCEPTION WHEN check_violation THEN RAISE NOTICE 'caught';
END$$;
ERROR: cannot start a subtransaction when there are unprocessed inval
messages
CONTEXT: SQL statement "ALTER DOMAIN d2 ADD CONSTRAINT d2_k CHECK (VALUE
>= -303)"

It also happens with DROP CONSTRAINT IF EXISTS of a constraint that
does not exist, with a domain NOT NULL constraint, when a plain SELECT
runs between the DROP and the SAVEPOINT, and inside a single DO block
outside a transaction block (DROP, then an inner BEGIN ... EXCEPTION
block). It does not happen if the subtransaction changes no catalogs,
if another catalog-changing statement runs at the top level between
the DROP and the SAVEPOINT, or after ALTER DOMAIN ... ADD CONSTRAINT
instead of DROP. Nothing is logged besides the error; after ROLLBACK
the session works normally.

Reproduced on 18.6 (REL_18_STABLE a85b5523e67) and master
158b6d5109f, macOS arm64, --enable-cassert. The first failing commit
is 3aafc030a53 ("Reduce memory consumption for pending invalidation
messages"), which added this check; its parent works.

As far as I can tell: AlterDomainDropConstraint() calls
performDeletion(), whose deleteOneObject() ends with
CommandCounterIncrement(), and only afterwards queues
CacheInvalidateHeapTuple() for the domain's pg_type row. That writes
nothing, so the command ID is not marked used, and the
CommandCounterIncrement() at the end of the statement does nothing.
The message stays in the top level's CurrentCmdInvalidMsgs until the
next subtransaction registers its own message, and
PrepareInvalidationState() raises the error.

Found with a query fuzzer.

Regards,
Alperen Keles