Backend crash (signal 11) in pg_trgm makesign() after ALTER TABLE ... SET STORAGE on a column with a gist_trgm_ops index

Hello,

We found that after *ALTER TABLE ... ALTER COLUMN ... SET STORAGE EXTENDED*
(or MAIN) on a text column that has a
gist_trgm_ops index, the next INSERT or non-HOT UPDATE into the table
crashes the backend with signal 11.
It reproduces every time on a stock installation.

*== Version ==*

PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2) on aarch64-unknown-linux-gnu,
compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit

Also reproduced on 17.11 and 16.15 (same Debian PGDG packages), and first
seen on Amazon Aurora PostgreSQL 16.11 and Amazon RDS PostgreSQL 16.13.

*== Platform ==*

Official postgres:18 Docker image, Debian GNU/Linux 13 (trixie), glibc 2.41
(Debian GLIBC 2.41-12+deb13u4)
Kernel: Linux 6.12.76-linuxkit aarch64 (Docker Desktop VM on Apple
Silicon), 11 CPUs, 8 GB RAM
Configuration: the image's defaults; nothing changed in postgresql.conf, no
extra start-up options.

*== Steps to reproduce (psql, attached as repro.sql) ==*

CREATE EXTENSION pg_trgm;
CREATE TABLE t (id int, s text);
INSERT INTO t SELECT g, 'item ' || g || ' ' || left(md5(g::text), 8) FROM
generate_series(1, 10000) g;
CREATE INDEX t_s_gist ON t USING gist (s gist_trgm_ops);
SELECT attstorage FROM pg_attribute WHERE attrelid = 't_s_gist'::regclass;
ALTER TABLE t ALTER COLUMN s SET STORAGE EXTENDED;
SELECT attstorage FROM pg_attribute WHERE attrelid = 't_s_gist'::regclass;
INSERT INTO t SELECT g, 'new item ' || g FROM generate_series(10001, 11000)
g;

*== Actual output ==*

psql (with \set VERBOSITY verbose):

CREATE EXTENSION
CREATE TABLE
INSERT 0 10000
CREATE INDEX
attstorage
------------
p
(1 row)

ALTER TABLE
attstorage
------------
x
(1 row)

Server closed the connection unexpectedly
This probably means the server terminated abnormally before or while
processing the request.
Connection to server was lost.

*Server log:*

LOG: client backend (PID 129) was terminated by signal 11: Segmentation
fault
DETAIL: Failed process was running: INSERT INTO t SELECT g, 'new item ' ||
g FROM generate_series(10001, 11000) g;
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing

*== Expected output ==*

The final INSERT should succeed (INSERT 0 1000). SET STORAGE EXTENDED does
not change the storage of a text
column (EXTENDED is already its default), and I would expect it to leave
the index column alone: the
index column's type is gtrgm, whose typstorage is 'p', and a newly created
index on the same column gets 'p'.

*== Observations ==*

- SET STORAGE MAIN crashes the same way (index column p -> m). SET STORAGE
PLAIN does not crash (p -> p).
- REINDEX or DROP + CREATE INDEX sets the index column back to 'p' and the
crash stops.
- With core's tsvector GiST opclass (key type gtsvector, typstorage 'p'),
SET STORAGE EXTENDED also changes
the index column to 'x', but inserts do not crash.

Backtrace on 16.13 with the PGDG debug symbols (postgresql-16-dbgsym):

#0 makesign (sign=..., a=0xaaaaf41f9498, siglen=12) at
contrib/pg_trgm/trgm_gist.c:109
len = 44914044
#1 gtrgm_penalty (fcinfo=...) at contrib/pg_trgm/trgm_gist.c:722
#3 gistpenalty () at src/backend/access/gist/gistutil.c:733
#4 gistchoose () at src/backend/access/gist/gistutil.c:458
#5 gistdoinsert () at src/backend/access/gist/gist.c:749
#6 gistinsert () at src/backend/access/gist/gist.c:184
#7 ExecInsertIndexTuples () at src/backend/executor/execIndexing.c:432

In another run, the crash was in unionkey() at
contrib/pg_trgm/trgm_gist.c:555.

Bytes of the key being inserted (gdb: x/12xb a, in makesign):

b9 01 20 20 31 20 20 32 20 20 70 20

The first byte (0xb9) is a 1-byte varlena header (length 92), followed by
the ARRKEY flag (0x01) and the
trigrams " 1", " 2", " p". Read with a 4-byte header, as the TRGM macros
do, the length is
(0x202001b9 >> 2) and the flag byte is 0x31, which matches len = 44914044
above.

*== Where this seems to come from ==*

SetIndexStorageProperties() in src/backend/commands/tablecmds.c (current
master) sets attstorage on every
index column whose indkey references the altered column, without comparing
the index column's type with
the table column's type. With attstorage 'x' or 'm' on the gtrgm column,
the new index tuple gets a short
varlena header, and gtrgm_decompress() passes it on unchanged (it uses
DatumGetTextPP).

I hope this helps.
Best,

*Thiago Bonfante*
Director of Engineering
<eric(at)subbase(dot)io>
<https://www.subbase.io/>
<+19546847379>
<https://www.instagram.com/subbase.io/>

thiago(at)subbase(dot) <thiago(at)subbase(dot)io>io
subbase.io <https://www.subbase.io/>
+55 (45) 98811 5410 <+5545988115410>
<https://www.linkedin.com/company/subbase/>
<https://www.subbase.io/>