BUG #14212: DB crash: system logger process was terminated by exception 0xFFFFFFFFwith exception 0xFFFFFFFF

The following bug has been logged on the website:

Bug reference: 14212
Logged by: Jie Long
Email address: jie(dot)long(at)pega(dot)com
PostgreSQL version: 9.3.13
Operating system: Windows 7
Description:

On our build system, the Postgres database will crash intermittently with
the following error:

2016-06-17 11:06:35 GMT 3724 0STATEMENT: select pzPVStream ,
pxCommitDateTime from engadmin.pr4_rule_file where pzInsKey = $1
2016-06-17 11:06:35 GMT 3724 0DEBUG: 00000: name: unnamed; blockState:
DEFAULT; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children:
2016-06-17 11:06:35 GMT 3724 0LOCATION: ShowTransactionStateRec,
src\backend\access\transam\xact.c:4513
2016-06-17 11:06:35 GMT 3724 0STATEMENT: select pzPVStream ,
pxCommitDateTime from engadmin.pr4_rule_file where pzInsKey = $1
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: CommitTransactionCommand
2016-06-17 11:06:35 GMT 1384 0LOCATION: finish_xact_command,
src\backend\tcop\postgres.c:2438
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: CommitTransaction
2016-06-17 11:06:35 GMT 1384 0LOCATION: ShowTransactionState,
src\backend\access\transam\xact.c:4475
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: reaping dead processes
2016-06-17 11:06:35 GMT 1092 0LOCATION: reaper,
src\backend\postmaster\postmaster.c:2660
2016-06-17 11:06:35 GMT 3724 0DEBUG: 00000: bind <unnamed> to <unnamed>
2016-06-17 11:06:35 GMT 3724 0LOCATION: exec_bind_message,
src\backend\tcop\postgres.c:1461
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: name: unnamed; blockState:
STARTED; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children:
2016-06-17 11:06:35 GMT 1384 0LOCATION: ShowTransactionStateRec,
src\backend\access\transam\xact.c:4513
2016-06-17 11:06:35 GMT 3724 0LOG: 00000: execute <unnamed>: select
pzPVStream , pxCommitDateTime from engadmin.pr4_rule_file where pzInsKey =
$1
2016-06-17 11:06:35 GMT 3724 0DETAIL: parameters: $1 = 'RULE-FILE-TEXT
WEBWB FUSIONCHARTS.UK!JS #20150211T112924.120 GMT'
2016-06-17 11:06:35 GMT 3724 0LOCATION: exec_execute_message,
src\backend\tcop\postgres.c:1935
2016-06-17 11:06:35 GMT 1092 0LOG: 00000: system logger process (PID 3024)
was terminated by exception 0xFFFFFFFF
2016-06-17 11:06:35 GMT 1092 0HINT: See C include file "ntstatus.h" for a
description of the hexadecimal value.
2016-06-17 11:06:35 GMT 1092 0LOCATION: LogChildExit,
src\backend\postmaster\postmaster.c:3387
2016-06-17 11:06:35 GMT 3724 0DEBUG: 00000: CommitTransactionCommand
2016-06-17 11:06:35 GMT 3724 0LOCATION: finish_xact_command,
src\backend\tcop\postgres.c:2438
2016-06-17 11:06:35 GMT 3724 0DEBUG: 00000: CommitTransaction
2016-06-17 11:06:35 GMT 3724 0LOCATION: ShowTransactionState,
src\backend\access\transam\xact.c:4475
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: parse <unnamed>: select
pzPVStream , pxCommitDateTime from engadmin.pr4_rule_property where
"pyruleavailable" not like $1 and "pxinsid" = $2 and pzInsKey like $3
order by pzInsKey
2016-06-17 11:06:35 GMT 1384 0LOCATION: exec_parse_message,
src\backend\tcop\postgres.c:1201
2016-06-17 11:06:35 GMT 1384 0STATEMENT: select pzPVStream ,
pxCommitDateTime from engadmin.pr4_rule_property where "pyruleavailable" not
like $1 and "pxinsid" = $2 and pzInsKey like $3 order by pzInsKey
2016-06-17 11:06:35 GMT 3724 0DEBUG: 00000: name: unnamed; blockState:
STARTED; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children:
2016-06-17 11:06:35 GMT 3724 0LOCATION: ShowTransactionStateRec,
src\backend\access\transam\xact.c:4513
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: StartTransactionCommand
2016-06-17 11:06:35 GMT 1384 0LOCATION: start_xact_command,
src\backend\tcop\postgres.c:2414
2016-06-17 11:06:35 GMT 1384 0STATEMENT: select pzPVStream ,
pxCommitDateTime from engadmin.pr4_rule_property where "pyruleavailable" not
like $1 and "pxinsid" = $2 and pzInsKey like $3 order by pzInsKey
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: StartTransaction
2016-06-17 11:06:35 GMT 1384 0LOCATION: ShowTransactionState,
src\backend\access\transam\xact.c:4475
2016-06-17 11:06:35 GMT 1384 0STATEMENT: select pzPVStream ,
pxCommitDateTime from engadmin.pr4_rule_property where "pyruleavailable" not
like $1 and "pxinsid" = $2 and pzInsKey like $3 order by pzInsKey
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: name: unnamed; blockState:
DEFAULT; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children:
2016-06-17 11:06:35 GMT 1384 0LOCATION: ShowTransactionStateRec,
src\backend\access\transam\xact.c:4513
2016-06-17 11:06:35 GMT 1384 0STATEMENT: select pzPVStream ,
pxCommitDateTime from engadmin.pr4_rule_property where "pyruleavailable" not
like $1 and "pxinsid" = $2 and pzInsKey like $3 order by pzInsKey
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: bind <unnamed> to <unnamed>
2016-06-17 11:06:35 GMT 1384 0LOCATION: exec_bind_message,
src\backend\tcop\postgres.c:1461
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: reaping dead processes
2016-06-17 11:06:35 GMT 1092 0LOCATION: reaper,
src\backend\postmaster\postmaster.c:2660
2016-06-17 11:06:35 GMT 1092 0LOG: 00000: background writer process (PID
3400) was terminated by exception 0xFFFFFFFF
2016-06-17 11:06:35 GMT 1092 0HINT: See C include file "ntstatus.h" for a
description of the hexadecimal value.
2016-06-17 11:06:35 GMT 1092 0LOCATION: LogChildExit,
src\backend\postmaster\postmaster.c:3387
2016-06-17 11:06:35 GMT 1384 0LOG: 00000: execute <unnamed>: select
pzPVStream , pxCommitDateTime from engadmin.pr4_rule_property where
"pyruleavailable" not like $1 and "pxinsid" = $2 and pzInsKey like $3
order by pzInsKey
2016-06-17 11:06:35 GMT 1384 0DETAIL: parameters: $1 = 'No', $2 =
'!PYISHYBRIDCLIENT', $3 = 'RULE-OBJ-PROPERTY %'
2016-06-17 11:06:35 GMT 1384 0LOCATION: exec_execute_message,
src\backend\tcop\postgres.c:1935
2016-06-17 11:06:35 GMT 1092 0LOG: 00000: terminating any other active
server processes
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3125
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
1384
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3224
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
1636
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3224
2016-06-17 11:06:35 GMT 1384 0WARNING: 57P00: terminating connection
because of crash of another server process
2016-06-17 11:06:35 GMT 1384 0DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.
2016-06-17 11:06:35 GMT 1384 0HINT: In a moment you should be able to
reconnect to the database and repeat your command.
2016-06-17 11:06:35 GMT 1384 0LOCATION: quickdie,
src\backend\tcop\postgres.c:2584
2016-06-17 11:06:35 GMT 1384 0STATEMENT: select pzPVStream ,
pxCommitDateTime from engadmin.pr4_rule_property where "pyruleavailable" not
like $1 and "pxinsid" = $2 and pzInsKey like $3 order by pzInsKey
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: shmem_exit(-1): 0 callbacks to
make
2016-06-17 11:06:35 GMT 1384 0LOCATION: shmem_exit,
src\backend\storage\ipc\ipc.c:212
2016-06-17 11:06:35 GMT 1384 0DEBUG: 00000: proc_exit(-1): 0 callbacks to
make
2016-06-17 11:06:35 GMT 1384 0LOCATION: proc_exit_prepare,
src\backend\storage\ipc\ipc.c:184
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
344
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3224
2016-06-17 11:06:35 GMT 1636 0WARNING: 57P00: terminating connection
because of crash of another server process
2016-06-17 11:06:35 GMT 1636 0DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.
2016-06-17 11:06:35 GMT 1636 0HINT: In a moment you should be able to
reconnect to the database and repeat your command.
2016-06-17 11:06:35 GMT 1636 0LOCATION: quickdie,
src\backend\tcop\postgres.c:2584
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
3664
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3224
2016-06-17 11:06:35 GMT 1636 0DEBUG: 00000: shmem_exit(-1): 0 callbacks to
make
2016-06-17 11:06:35 GMT 1636 0LOCATION: shmem_exit,
src\backend\storage\ipc\ipc.c:212
2016-06-17 11:06:35 GMT 1636 0DEBUG: 00000: proc_exit(-1): 0 callbacks to
make
2016-06-17 11:06:35 GMT 1636 0LOCATION: proc_exit_prepare,
src\backend\storage\ipc\ipc.c:184
2016-06-17 11:06:35 GMT 344 0WARNING: 57P00: terminating connection because
of crash of another server process
2016-06-17 11:06:35 GMT 344 0DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.
2016-06-17 11:06:35 GMT 344 0HINT: In a moment you should be able to
reconnect to the database and repeat your command.
2016-06-17 11:06:35 GMT 344 0LOCATION: quickdie,
src\backend\tcop\postgres.c:2584
2016-06-17 11:06:35 GMT 344 0DEBUG: 00000: shmem_exit(-1): 0 callbacks to
make
2016-06-17 11:06:35 GMT 344 0LOCATION: shmem_exit,
src\backend\storage\ipc\ipc.c:212
2016-06-17 11:06:35 GMT 344 0DEBUG: 00000: proc_exit(-1): 0 callbacks to
make
2016-06-17 11:06:35 GMT 344 0LOCATION: proc_exit_prepare,
src\backend\storage\ipc\ipc.c:184
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
1420
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3224
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
2220
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3224
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: kill(2220,3) failed: No such
process
2016-06-17 11:06:35 GMT 1092 0LOCATION: signal_child,
src\backend\postmaster\postmaster.c:3673
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
3724
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3224
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: kill(3724,3) failed: No such
process
2016-06-17 11:06:35 GMT 1092 0LOCATION: signal_child,
src\backend\postmaster\postmaster.c:3673
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
2408
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3266
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: kill(2408,3) failed: No such
process
2016-06-17 11:06:35 GMT 1092 0LOCATION: signal_child,
src\backend\postmaster\postmaster.c:3673
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
3780
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3278
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: kill(3780,3) failed: No such
process
2016-06-17 11:06:35 GMT 1092 0LOCATION: signal_child,
src\backend\postmaster\postmaster.c:3673
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
2268
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3302
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: kill(2268,3) failed: No such
process
2016-06-17 11:06:35 GMT 1092 0LOCATION: signal_child,
src\backend\postmaster\postmaster.c:3673
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: sending SIGQUIT to process
3052
2016-06-17 11:06:35 GMT 1092 0LOCATION: HandleChildCrash,
src\backend\postmaster\postmaster.c:3332
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: kill(3052,3) failed: No such
process
2016-06-17 11:06:35 GMT 1092 0LOCATION: signal_child,
src\backend\postmaster\postmaster.c:3673
2016-06-17 11:06:35 GMT 1092 0LOG: 00000: statistics collector process (PID
3052) was terminated by exception 0xFFFFFFFF
2016-06-17 11:06:35 GMT 1092 0HINT: See C include file "ntstatus.h" for a
description of the hexadecimal value.
2016-06-17 11:06:35 GMT 1092 0LOCATION: LogChildExit,
src\backend\postmaster\postmaster.c:3387
2016-06-17 11:06:35 GMT 1092 0DEBUG: 00000: reaping dead processes
2016-06-17 11:06:35 GMT 1092 0LOCATION: reaper,
src\backend\postmaster\postmaster.c:2660
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: find_in_dynamic_libpath: trying
"e:/postgres/lib/pg_stat_statements"
2016-06-17 11:06:35 GMT 2480 0LOCATION: find_in_dynamic_libpath,
src\backend\utils\fmgr\dfmgr.c:638
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: find_in_dynamic_libpath: trying
"E:/postgres/share/pljava/pg_stat_statements"
2016-06-17 11:06:35 GMT 2480 0LOCATION: find_in_dynamic_libpath,
src\backend\utils\fmgr\dfmgr.c:638
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: find_in_dynamic_libpath: trying
"e:/postgres/lib/pg_stat_statements.dll"
2016-06-17 11:06:35 GMT 2480 0LOCATION: find_in_dynamic_libpath,
src\backend\utils\fmgr\dfmgr.c:638
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: loaded library
"pg_stat_statements"
2016-06-17 11:06:35 GMT 2480 0LOCATION: load_libraries,
src\backend\utils\init\miscinit.c:1370
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: logger shutting down
2016-06-17 11:06:35 GMT 2480 0LOCATION: SysLoggerMain,
src\backend\postmaster\syslogger.c:506
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: shmem_exit(0): 0 callbacks to
make
2016-06-17 11:06:35 GMT 2480 0LOCATION: shmem_exit,
src\backend\storage\ipc\ipc.c:212
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: proc_exit(0): 0 callbacks to
make
2016-06-17 11:06:35 GMT 2480 0LOCATION: proc_exit_prepare,
src\backend\storage\ipc\ipc.c:184
2016-06-17 11:06:35 GMT 2480 0DEBUG: 00000: exit(0)
2016-06-17 11:06:35 GMT 2480 0LOCATION: proc_exit,
src\backend\storage\ipc\ipc.c:135

Version: "PostgreSQL 9.3.13, compiled by Visual C++ build 1600, 64-bit"

How you installed PostgreSQL
Downloaded the EnterpriseDB One-click installer?

Yes

Windows Version: Windows 7 Enterprise Service Pack 1 (Version 6.1 Build
7601: Service Pack 1)

We have Cygwin on the environment path. But we used dependency walker and
verified the Postgres.exe does not have any DLL dependency on Cygwin.

How did we start Postgres:

We started Postgres from command line instead of a service

Postgres Log/Error details:

We encounter the error intermittently. It happens when:
Our application backend agents (from different VM) are running some elastic
search indexing--reading blobs from database columns;

And tomcat (on the same VM) is started and some simple REST GET calls are
trying to access this application's front end with tomcat initiated database
connections

For questions about any kind of error:

JDBC Version: postgresql-9.3-1104.jdbc41.jar

Java Version:
1.7.0_65
Java(TM) SE Runtime Environment (build 1.7.0_b6-b19)
Java HotSpot(TM) 64-Bit Server VM ( build 24.65-b04, mixed mode)

Tomcat Version: Tomcat 7.0.53