BUG #14238: crate index hang

The following bug has been logged on the website:

Bug reference: 14238
Logged by: Guang-Dih Lei
Email address: guang-dih(dot)lei(at)nasa(dot)gov
PostgreSQL version: 9.5.3
Operating system: Linux - CentOS release 6.8
Description:

I have a table with around 108 millions records. After I issued the
following 'create index' command, it just hang and can never finish:

CREATE INDEX i_geo_box ON geometricobject USING gist (geo_box);

Here is my schema on the table:
mirador=# \d geometricobject;
Table "public.geometricobject"
Column | Type | Modifiers

-------------+---------+-----------------------------------------------------------------------
geometricid | integer | not null default
nextval('geometricobject_geometricid_seq'::regclass)
spatialid | integer | not null
geo_box | box | not null
Indexes:
"geometricobject_pkey" PRIMARY KEY, btree (geometricid)
"i_geo_spatialid" btree (spatialid)
Foreign-key constraints:
"fk_geoobj_ref_spatobj" FOREIGN KEY (spatialid) REFERENCES
spatialobject(spatialid) ON UPDATE CASCADE ON DELETE CASCADE

We put a strace on the create index process, the output just stop after
around 30 minutes. Here is the final output of the strace:

open("base/974126235/1338007928_fsm", O_RDWR) = -1 ENOENT (No such file or
directory)
lseek(53, 0, SEEK_END) = 527228928
write(53,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
8192) = 8192
lseek(51, 761561088, SEEK_SET) = 761561088
read(51, "\325l\3\0\30\0\0\0\246\233\304
\2602>\300\315\314\314\314\314\230e(at)\0\0\0\0\0\0\0\0\r\0joK\0(\0=\n\327\243p\2752(at)\0\0\0\0\0pf@\315\314\314\314\314\314\364\277\24\256G\341z\254e(at)\r\0ro\34\0(\0\217\302\365(\\\17*(at)\0\0\0\0\0pf@\327\243p=\n\327\35\300)\\\217\302\365\220e(at)\r\0toO\0(\0\217\302\365(\\\17*(at)\0\0\0\0"...,
8192) = 8192
lseek(51, 765108224, SEEK_SET) = 765108224
read(51, "\\o\2\0\30\0\0\0\246\233\304
\2602>\300\315\314\314\314\314\230e(at)\0\0\0\0\0\0\0\0\r\0\304r\20\0(\0\270\36\205\353Q85(at)\0\0\0\0\0pf@\232\231\231\231\231\231\371?{\24\256G\341\312e(at)\r\0\305rQ\0(\0{\24\256G\341z-(at)\0\0\0\0\0pf@\37\205\353Q\270\36\30\300q=\n\327\243(at)f@\r\0\306r6\0(\0\217\302\365(\\\3174(at)\0\0\0\0"...,
8192) = 8192
lseek(51, 233537536, SEEK_SET) = 233537536
read(51, "$?\3\0\30\0\0\0\246\233\304
\2602>\300\315\314\314\314\314\230e(at)\0\0\0\0\0\0\0\0\r\0\314v:\0(\0\10\254\34Zd{)@\0\0\0\0\0\200f(at)%\6\201\225C\213\354?P\215\227n\22\3f@\r\0\321v2\0(\0\24\256G\341z\3243(at)\0\0\0\0\0pf@q=\n\327\243p\315\277H\341z\24\2567f(at)\r\0\325v0\0(\0\327\243p=\nW5(at)\0\0\0\0"...,
8192) = 8192
open("base/974126235/1338007928_fsm", O_RDWR) = -1 ENOENT (No such file or
directory)
lseek(53, 0, SEEK_END) = 527237120
write(53,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
8192) = 8192
open("base/974126235/1338007928_fsm", O_RDWR) = -1 ENOENT (No such file or
directory)
lseek(53, 0, SEEK_END) = 527245312
write(53,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
8192) = 8192
lseek(51, 669286400, SEEK_SET) = 669286400
read(51, "J!\1\0\30\0\0\0\246\233\304
\2602>\300\315\314\314\314\314\230e(at)\0\0\0\0\0\0\0\0\r\0\255z\23\0(\0\226C\213l\347\273#(at)u\223\30\4Vje@\n\327\243p=J\"@\347\373\251\361\322ae(at)\r\0\260z\35\0(\0;\337O\215\227\36@@\276\237\32/\335(at)f@\22\203\300\312\241\205?(at)\f\2+\207\26\355e@\r\0\260z#\0(\0\217\302\365(\\\357;@\254\34Zd"...,
8192) = 8192

I am not sure about the missing _fsm file message. However, we saw the
message right from the beginning of strace output.

I know the index is probably being created with filepath 1338007928*. Those
files timestamp stop being updated after the strace output stop:

$:/db/pgsql/9.5/data/base/974126235$ ls -ltr 1338007928*
-rw------- 1 postgres postgres 1073741824 Jul 7 19:27 1338007928.1
-rw------- 1 postgres postgres 1073741824 Jul 7 19:27 1338007928.2
-rw------- 1 postgres postgres 1073741824 Jul 7 19:27 1338007928.3
-rw------- 1 postgres postgres 527253504 Jul 7 19:28 1338007928.4
-rw------- 1 postgres postgres 1073741824 Jul 7 19:32 1338007928

The database server is on a host with 32GB RAM. My shared_buffers is set at
8GB. I have tried both 'posix' and 'sysv' on 'dynamic_shared_memory_type'
parameter. Neither one can have the index creation process completed.

We have the same database on another host with Postgres-9.1 which we don't
have problem creating that index. Since 9.1 will be out of support soon, we
are migrating it to a new 9.5 server.

I appreciate if anyone can help. Thanks.