pgsql: Background worker processes

Background worker processes

Background workers are postmaster subprocesses that run arbitrary
user-specified code. They can request shared memory access as well as
backend database connections; or they can just use plain libpq frontend
database connections.

Modules listed in shared_preload_libraries can register background
workers in their _PG_init() function; this is early enough that it's not
necessary to provide an extra GUC option, because the necessary extra
resources can be allocated early on. Modules can install more than one
bgworker, if necessary.

Care is taken that these extra processes do not interfere with other
postmaster tasks: only one such process is started on each ServerLoop
iteration. This means a large number of them could be waiting to be
started up and postmaster is still able to quickly service external
connection requests. Also, shutdown sequence should not be impacted by
a worker process that's reasonably well behaved (i.e. promptly responds
to termination signals.)

The current implementation lets worker processes specify their start
time, i.e. at what point in the server startup process they are to be
started: right after postmaster start (in which case they mustn't ask
for shared memory access), when consistent state has been reached
(useful during recovery in a HOT standby server), or when recovery has
terminated (i.e. when normal backends are allowed).

In case of a bgworker crash, actions to take depend on registration
data: if shared memory was requested, then all other connections are
taken down (as well as other bgworkers), just like it were a regular
backend crashing. The bgworker itself is restarted, too, within a
configurable timeframe (which can be configured to be never).

More features to add to this framework can be imagined without much
effort, and have been discussed, but this seems good enough as a useful
unit already.

An elementary sample module is supplied.

Author: Álvaro Herrera

This patch is loosely based on prior patches submitted by KaiGai Kohei,
and unsubmitted code by Simon Riggs.

Reviewed by: KaiGai Kohei, Markus Wanner, Andres Freund,
Heikki Linnakangas, Simon Riggs, Amit Kapila

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/da07a1e856511dca59cbb1357616e26baa64428e

Modified Files
--------------
contrib/Makefile | 3 +-
contrib/worker_spi/Makefile | 14 +
contrib/worker_spi/worker_spi.c | 263 ++++++++
doc/src/sgml/bgworker.sgml | 146 +++++
doc/src/sgml/filelist.sgml | 1 +
doc/src/sgml/postgres.sgml | 1 +
src/backend/postmaster/postmaster.c | 1165 ++++++++++++++++++++++++++++++++--
src/backend/storage/lmgr/proc.c | 43 +-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/miscinit.c | 6 +-
src/backend/utils/init/postinit.c | 13 +
src/backend/utils/misc/guc.c | 15 +-
src/include/miscadmin.h | 1 +
src/include/postmaster/bgworker.h | 104 ++++
src/include/postmaster/postmaster.h | 2 +
src/include/storage/proc.h | 2 +
16 files changed, 1690 insertions(+), 90 deletions(-)