Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

On Tue, Jul 28, 2020 at 11:10 AM Justin Pryzby <pryzby(at)telsasoft(dot)com> wrote:
>
> An internal server aborted last night while running a maintenance script. I
> reproduced this easily running the crashing command in a loop, and verified
> this is a live issue on REL_13_STABLE (dc6f2fb43).
> REINDEX INDEX pg_class_tblspc_relfilenode_index
>
> It looks like this crashed once before, and I didn't notice until now:
> Fri Jun 26 22:30:29 CDT 2020: pg_shdescription: pg_toast.pg_toast_2396_index(reindex toast)...
> psql: error: could not connect to server: server closed the connection unexpectedly
>
> #3 0x0000000000afde98 in comparetup_index_btree (a=0x201fa58, b=0x201fa10, state=0x20147b0) at tuplesort.c:4251
> 4251 Assert(false);
> (gdb) l
> 4246 if (pos1 != pos2)
> 4247 return (pos1 < pos2) ? -1 : 1;
> 4248 }
> 4249
> 4250 /* ItemPointer values should never be equal */
> 4251 Assert(false);
> 4252
> 4253 return 0;
> 4254 }
> 4255
>
> #3 0x0000000000afde98 in comparetup_index_btree (a=0x201fa58, b=0x201fa10, state=0x20147b0) at tuplesort.c:4251
> sortKey = 0x2014d60
> tuple1 = 0x20189d8
> tuple2 = 0x2018898
> keysz = 2
> tupDes = 0x7f48996b3790
> equal_hasnull = false
> nkey = 3
> compare = 0
> datum1 = 67999603
> datum2 = 67999603
> isnull1 = false
> isnull2 = false
> __func__ = "comparetup_index_btree"
>
> (gdb) p *tuple1
> $2 = {t_tid = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 43}, t_info = 16}
> (gdb) p *tuple2
> $3 = {t_tid = {ip_blkid = {bi_hi = 0, bi_lo = 0}, ip_posid = 43}, t_info = 16}
>
> (gdb) p *sortKey
> $5 = {ssup_cxt = 0x40, ssup_collation = 0, ssup_reverse = false, ssup_nulls_first = false, ssup_attno = 0, ssup_extra = 0x0, comparator = 0x7f7f7f7f7f7f7f7f, abbreviate = 127, abbrev_converter = 0x7f7f7f7f7f7f7f7f,
> abbrev_abort = 0x7f7f7f7f7f7f7f7f, abbrev_full_comparator = 0x7f7f7f7f7f7f7f7f}
>
> (gdb) p *tupDes
> $6 = {natts = 2, tdtypeid = 0, tdtypmod = -1, tdrefcount = 1, constr = 0x0, attrs = 0x7f48996b37a8}
>
> Core was generated by `postgres: postgres sentinel [local] REINDEX '.
>
> (gdb) bt
> #0 0x00007f489853d1f7 in raise () from /lib64/libc.so.6
> #1 0x00007f489853e8e8 in abort () from /lib64/libc.so.6
> #2 0x0000000000aafff7 in ExceptionalCondition (conditionName=0xccd0dc "false", errorType=0xccc327 "FailedAssertion", fileName=0xccc2fd "tuplesort.c", lineNumber=4251) at assert.c:67
> #3 0x0000000000afde98 in comparetup_index_btree (a=0x201fa58, b=0x201fa10, state=0x20147b0) at tuplesort.c:4251
> #4 0x0000000000af1d5e in qsort_tuple (a=0x201fa10, n=18, cmp_tuple=0xafcf21 <comparetup_index_btree>, state=0x20147b0) at qsort_tuple.c:140
> #5 0x0000000000af2104 in qsort_tuple (a=0x201f710, n=50, cmp_tuple=0xafcf21 <comparetup_index_btree>, state=0x20147b0) at qsort_tuple.c:191
> #6 0x0000000000af2104 in qsort_tuple (a=0x201cc38, n=544, cmp_tuple=0xafcf21 <comparetup_index_btree>, state=0x20147b0) at qsort_tuple.c:191
> #7 0x0000000000af8056 in tuplesort_sort_memtuples (state=0x20147b0) at tuplesort.c:3490
> #8 0x0000000000af51a9 in tuplesort_performsort (state=0x20147b0) at tuplesort.c:1985
> #9 0x0000000000529418 in _bt_leafbuild (btspool=0x1f784e0, btspool2=0x0) at nbtsort.c:553
> #10 0x0000000000528f9c in btbuild (heap=0x1fb5758, index=0x7f48996b3460, indexInfo=0x1f77a48) at nbtsort.c:333
> #11 0x00000000005adcb3 in index_build (heapRelation=0x1fb5758, indexRelation=0x7f48996b3460, indexInfo=0x1f77a48, isreindex=true, parallel=true) at index.c:2903
> #12 0x00000000005aec6b in reindex_index (indexId=3455, skip_constraint_checks=false, persistence=112 'p', options=2) at index.c:3539
> #13 0x0000000000692583 in ReindexIndex (indexRelation=0x1f54840, options=0, concurrent=false) at indexcmds.c:2466
> #14 0x0000000000932e36 in standard_ProcessUtility (pstmt=0x1f54960, queryString=0x1f53d90 "REINDEX INDEX pg_class_tblspc_relfilenode_index", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x1f54c40,
> qc=0x7ffde023bf80) at utility.c:929
> #15 0x000000000093241f in ProcessUtility (pstmt=0x1f54960, queryString=0x1f53d90 "REINDEX INDEX pg_class_tblspc_relfilenode_index", context=PROCESS_UTILITY_TOPLEVEL, params=0x0, queryEnv=0x0, dest=0x1f54c40, qc=0x7ffde023bf80)
> at utility.c:524
> #16 0x0000000000931288 in PortalRunUtility (portal=0x1fb5ac0, pstmt=0x1f54960, isTopLevel=true, setHoldSnapshot=false, dest=0x1f54c40, qc=0x7ffde023bf80) at pquery.c:1157
> #17 0x00000000009314a7 in PortalRunMulti (portal=0x1fb5ac0, isTopLevel=true, setHoldSnapshot=false, dest=0x1f54c40, altdest=0x1f54c40, qc=0x7ffde023bf80) at pquery.c:1303
> #18 0x00000000009309bc in PortalRun (portal=0x1fb5ac0, count=9223372036854775807, isTopLevel=true, run_once=true, dest=0x1f54c40, altdest=0x1f54c40, qc=0x7ffde023bf80) at pquery.c:779
> #19 0x000000000092ab5b in exec_simple_query (query_string=0x1f53d90 "REINDEX INDEX pg_class_tblspc_relfilenode_index") at postgres.c:1239
> #20 0x000000000092eb82 in PostgresMain (argc=1, argv=0x1f7db80, dbname=0x1f509d8 "sentinel", username=0x1f7daa0 "pryzbyj") at postgres.c:4315
> #21 0x000000000087f098 in BackendRun (port=0x1f75a80) at postmaster.c:4523
> #22 0x000000000087e888 in BackendStartup (port=0x1f75a80) at postmaster.c:4215
> #23 0x000000000087ae95 in ServerLoop () at postmaster.c:1727
> #24 0x000000000087a76c in PostmasterMain (argc=5, argv=0x1f4e8e0) at postmaster.c:1400
> #25 0x00000000007823f3 in main (argc=5, argv=0x1f4e8e0) at main.c:210
>
> This appears to be an issue with d2d8a229b (Incremental Sort), so I will add
> at: https://wiki.postgresql.org/wiki/PostgreSQL_13_Open_Items

Is that assumption largely based on the incremental sort patch
refactoring tuplesort.c a bit? I haven't looked at it much at all, but
I'm wondering if the issue could also be related to the btree
duplicates changes in 13 given that we're looking at
comparetup_index_btree and the datums are equal.

James