Re: Backend crash (signal 11) in pg_trgm makesign() after ALTER TABLE ... SET STORAGE on a column with a gist_trgm_ops index

Hi Andrey,

Thanks for the quick response.

Best,
Thiago

On Fri, Oct 2, 2026 at 4:58 AM Andrey Rachitskiy <pl0h0yp1(at)gmail(dot)com> wrote:

>
> пт, 2 окт. 2026 г. в 11:07, Thiago Bonfante <thiago(at)subbase(dot)io>:
>
>> Hello,
>>
>> We found that after *ALTER TABLE ... ALTER COLUMN ... SET STORAGE
>> EXTENDED* (or MAIN) on a text column that has a
>> gist_trgm_ops index, the next INSERT or non-HOT UPDATE into the table
>> crashes the backend with signal 11.
>> It reproduces every time on a stock installation.
>>
>> *== Version ==*
>>
>> PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2) on aarch64-unknown-linux-gnu,
>> compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
>>
>> Also reproduced on 17.11 and 16.15 (same Debian PGDG packages), and
>> first seen on Amazon Aurora PostgreSQL 16.11 and Amazon RDS PostgreSQL
>> 16.13.
>>
>> *== Platform ==*
>>
>> Official postgres:18 Docker image, Debian GNU/Linux 13 (trixie), glibc
>> 2.41 (Debian GLIBC 2.41-12+deb13u4)
>> Kernel: Linux 6.12.76-linuxkit aarch64 (Docker Desktop VM on Apple
>> Silicon), 11 CPUs, 8 GB RAM
>> Configuration: the image's defaults; nothing changed in postgresql.conf,
>> no extra start-up options.
>>
>> *== Steps to reproduce (psql, attached as repro.sql) ==*
>>
>> CREATE EXTENSION pg_trgm;
>> CREATE TABLE t (id int, s text);
>> INSERT INTO t SELECT g, 'item ' || g || ' ' || left(md5(g::text), 8) FROM
>> generate_series(1, 10000) g;
>> CREATE INDEX t_s_gist ON t USING gist (s gist_trgm_ops);
>> SELECT attstorage FROM pg_attribute WHERE attrelid = 't_s_gist'::regclass;
>> ALTER TABLE t ALTER COLUMN s SET STORAGE EXTENDED;
>> SELECT attstorage FROM pg_attribute WHERE attrelid = 't_s_gist'::regclass;
>> INSERT INTO t SELECT g, 'new item ' || g FROM generate_series(10001,
>> 11000) g;
>>
>> *== Actual output ==*
>>
>> psql (with \set VERBOSITY verbose):
>>
>> CREATE EXTENSION
>> CREATE TABLE
>> INSERT 0 10000
>> CREATE INDEX
>> attstorage
>> ------------
>> p
>> (1 row)
>>
>> ALTER TABLE
>> attstorage
>> ------------
>> x
>> (1 row)
>>
>> Server closed the connection unexpectedly
>> This probably means the server terminated abnormally before or while
>> processing the request.
>> Connection to server was lost.
>>
>> *Server log:*
>>
>> LOG: client backend (PID 129) was terminated by signal 11: Segmentation
>> fault
>> DETAIL: Failed process was running: INSERT INTO t SELECT g, 'new item '
>> || g FROM generate_series(10001, 11000) g;
>> LOG: terminating any other active server processes
>> LOG: all server processes terminated; reinitializing
>>
>> *== Expected output ==*
>>
>> The final INSERT should succeed (INSERT 0 1000). SET STORAGE EXTENDED
>> does not change the storage of a text
>> column (EXTENDED is already its default), and I would expect it to leave
>> the index column alone: the
>> index column's type is gtrgm, whose typstorage is 'p', and a newly
>> created index on the same column gets 'p'.
>>
>> *== Observations ==*
>>
>> - SET STORAGE MAIN crashes the same way (index column p -> m). SET
>> STORAGE PLAIN does not crash (p -> p).
>> - REINDEX or DROP + CREATE INDEX sets the index column back to 'p' and
>> the crash stops.
>> - With core's tsvector GiST opclass (key type gtsvector, typstorage 'p'),
>> SET STORAGE EXTENDED also changes
>> the index column to 'x', but inserts do not crash.
>>
>> Backtrace on 16.13 with the PGDG debug symbols (postgresql-16-dbgsym):
>>
>> #0 makesign (sign=..., a=0xaaaaf41f9498, siglen=12) at
>> contrib/pg_trgm/trgm_gist.c:109
>> len = 44914044
>> #1 gtrgm_penalty (fcinfo=...) at contrib/pg_trgm/trgm_gist.c:722
>> #3 gistpenalty () at src/backend/access/gist/gistutil.c:733
>> #4 gistchoose () at src/backend/access/gist/gistutil.c:458
>> #5 gistdoinsert () at src/backend/access/gist/gist.c:749
>> #6 gistinsert () at src/backend/access/gist/gist.c:184
>> #7 ExecInsertIndexTuples () at src/backend/executor/execIndexing.c:432
>>
>> In another run, the crash was in unionkey() at
>> contrib/pg_trgm/trgm_gist.c:555.
>>
>> Bytes of the key being inserted (gdb: x/12xb a, in makesign):
>>
>> b9 01 20 20 31 20 20 32 20 20 70 20
>>
>> The first byte (0xb9) is a 1-byte varlena header (length 92), followed by
>> the ARRKEY flag (0x01) and the
>> trigrams " 1", " 2", " p". Read with a 4-byte header, as the TRGM macros
>> do, the length is
>> (0x202001b9 >> 2) and the flag byte is 0x31, which matches len = 44914044
>> above.
>>
>> *== Where this seems to come from ==*
>>
>> SetIndexStorageProperties() in src/backend/commands/tablecmds.c (current
>> master) sets attstorage on every
>> index column whose indkey references the altered column, without
>> comparing the index column's type with
>> the table column's type. With attstorage 'x' or 'm' on the gtrgm column,
>> the new index tuple gets a short
>> varlena header, and gtrgm_decompress() passes it on unchanged (it uses
>> DatumGetTextPP).
>>
>> I hope this helps.
>> Best,
>>
>>
>>
>> *Thiago Bonfante*
>> Director of Engineering
>> <eric(at)subbase(dot)io>
>> <https://www.subbase.io/>
>> <+19546847379>
>> <https://www.instagram.com/subbase.io/>
>>
>> thiago(at)subbase(dot) <thiago(at)subbase(dot)io>io
>> subbase.io <https://www.subbase.io/>
>> +55 (45) 98811 5410 <+5545988115410>
>> <https://www.linkedin.com/company/subbase/>
>> <https://www.subbase.io/>
>>
>
> Hi, Thiago!
>
> Thanks for the report!
>
> ALTER TABLE ... SET STORAGE updates simple index columns through
> SetIndexStorageProperties(). That copied the heap attstorage even when
> ConstructTupleDescriptor() had replaced the index column type with the
> opclass STORAGE type.
>
> gist_trgm_ops stores gtrgm, whose typstorage is PLAIN. After SET STORAGE
> EXTENDED the gist attstorage became EXTENDED. Later inserts packed short
> varlena headers. TRGM macros use VARSIZE on a four-byte header and the
> backend crashed.
>
> Skip the catalog update when the index column type differs from the heap
> column. A btree index on the same column still receives the new setting.
> SET COMPRESSION uses the same helper.
>
> --
> Regards,
> Rachitskiy Andrey
>