Skip site navigation (1) Skip section navigation (2)

Re: [HACKERS] loading libraries on Postmaster startup

From: Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us>
To: Joe Conway <mail(at)joeconway(dot)com>
Cc: "Patches (PostgreSQL)" <pgsql-patches(at)postgresql(dot)org>,Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Subject: Re: [HACKERS] loading libraries on Postmaster startup
Date: 2003-03-17 17:28:52
Message-ID: 200303171728.h2HHSqM27622@candle.pha.pa.us (view raw or flat)
Thread:
Lists: pgsql-hackerspgsql-patches
Your patch has been added to the PostgreSQL unapplied patches list at:

	http://momjian.postgresql.org/cgi-bin/pgpatches

I will try to apply it within the next 48 hours.

---------------------------------------------------------------------------


Joe Conway wrote:
> Joe Conway wrote:
> > Tom Lane wrote:
> >> Joe Conway <mail(at)joeconway(dot)com> writes:
> >>> In my testing with PL/R, it reduces the first call to a PL/R function 
> >>> (after connecting) from almost 2 seconds, down to about 8 ms.
> >>
> >> Hm, pretty significant.  Can you measure any per-fork cost (ie, the loss
> >> incurred by children that don't use PL/R)?  Is there any measurable
> >> benefit for our other PLs (plperl etc)?
> > 
> > Here's what I got:
> > 
> > 10000 connect/disconnect in tight loop
> > ----------------------------------------------------------
> > condition        time        top
> > ----------------------------------------------------------
> > with no preload        87 seconds    ~10% CPU, ~2.2 MB
> > with plr preload    133 seconds    ~10% CPU, ~13 MB
> > with plperl preload    92 seconds    ~10% CPU, ~3.2 MB
> > with pltcl preload    88 seconds    ~10% CPU, ~2.3 MB
> > with plpython preload    93 seconds    ~10% CPU, ~2.3 MB
> > 
> > 
> > 1000 connect/"select some_simple_func()"/disconnect in tight loop
> > ------------------------------------------------------------------
> > condition            time            top
> > ------------------------------------------------------------------
> > plr-func without preload    739 seconds    ~60% CPU, ~13 MB
> > plr-func with preload        26 seconds    ~10% CPU, ~13 MB
> > plperl-func without preload    46 seconds    ~4% CPU, ~3.2 MB
> > plperl-func with preload    33 seconds    ~3% CPU, ~3.2 MB
> > pltcl-func without preload    22 seconds    ~5% CPU, ~2.3 MB
> > pltcl-func with preload        17 seconds    ~4% CPU, ~2.3 MB
> > plpython-func without preload    33 seconds    ~4% CPU, ~2.3 MB
> > plpython-func with preload    31 seconds    ~4% CPU, ~2.3 MB
> > 
> 
> Here's an updated copy of the patch. The original patch was failing due 
> to code drift in cvs.
> 
> Peter's suggestion (__attribute__((constructor))) is interesting, but it 
> appears to be a gcc specific extension, and hence non-portable.
> 
> I'd still like to see this applied if there are no other objections. 
>  From the above info it is apparent that the benefit outweighs the cost 
> for plperl and pltcl (although not as significantly as for plr).
> 
> Joe

> Index: doc/src/sgml/runtime.sgml
> ===================================================================
> RCS file: /opt/src/cvs/pgsql-server/doc/src/sgml/runtime.sgml,v
> retrieving revision 1.169
> diff -c -r1.169 runtime.sgml
> *** doc/src/sgml/runtime.sgml	19 Feb 2003 04:06:28 -0000	1.169
> --- doc/src/sgml/runtime.sgml	27 Feb 2003 18:46:39 -0000
> ***************
> *** 1781,1786 ****
> --- 1781,1815 ----
>        </varlistentry>
>   
>        <varlistentry>
> +       <term><varname>PRELOAD_LIBRARIES</varname> (<type>string</type>)</term>
> +       <indexterm><primary>preload_libraries</></>
> +       <listitem>
> +        <para>
> +         This variable specifies one or more shared libraries that are to be
> + 	preloaded at Postmaster start. An initialization function can also be
> + 	optionally specified by adding a colon followed by the name of the
> + 	initialization function after the library name. For example
> + 	<literal>'$libdir/mylib:init_mylib'</literal> would cause <literal>mylib</>
> +     to be preloaded and <literal>init_mylib</> to be executed. If more than
> + 	one library is to be loaded, they must be delimited with a comma.
> +        </para>
> + 
> +        <para>
> +         If <literal>mylib</> is not found, the postmaster will fail to start.
> + 	However, if <literal>init_mylib</> is not found, <literal>mylib</> will
> + 	still be preloaded without executing the initialization function.
> +        </para>
> + 
> +        <para>
> +         By preloading a shared library (and initializing it if applicable),
> + 	the library startup time is avoided when the library is used later in a
> + 	specific backend. However there is a cost in terms of memory duplication
> + 	as every backend is forked, whether or not the library is used.
> +        </para>
> +       </listitem>
> +      </varlistentry>
> + 
> +      <varlistentry>
>         <term><varname>REGEX_FLAVOR</varname> (<type>string</type>)</term>
>         <indexterm><primary>regular expressions</></>
>         <listitem>
> Index: src/backend/postmaster/postmaster.c
> ===================================================================
> RCS file: /opt/src/cvs/pgsql-server/src/backend/postmaster/postmaster.c,v
> retrieving revision 1.307
> diff -c -r1.307 postmaster.c
> *** src/backend/postmaster/postmaster.c	23 Feb 2003 04:48:19 -0000	1.307
> --- src/backend/postmaster/postmaster.c	27 Feb 2003 18:46:39 -0000
> ***************
> *** 205,210 ****
> --- 205,212 ----
>   bool		Log_connections = false;
>   bool		Db_user_namespace = false;
>   
> + /* list of library:init-function to be preloaded */
> + char       *preload_libraries_string = NULL;
>   
>   /* Startup/shutdown state */
>   static pid_t StartupPID = 0,
> ***************
> *** 644,649 ****
> --- 646,658 ----
>   	if (EnableSSL)
>   		secure_initialize();
>   #endif
> + 
> + 	/*
> + 	 * process any libraries that should be preloaded and
> + 	 * optionally pre-initialized
> + 	 */
> + 	if (preload_libraries_string)
> + 		process_preload_libraries(preload_libraries_string);
>   
>   	/*
>   	 * Fork away from controlling terminal, if -S specified.
> Index: src/backend/utils/init/miscinit.c
> ===================================================================
> RCS file: /opt/src/cvs/pgsql-server/src/backend/utils/init/miscinit.c,v
> retrieving revision 1.100
> diff -c -r1.100 miscinit.c
> *** src/backend/utils/init/miscinit.c	27 Jan 2003 00:51:06 -0000	1.100
> --- src/backend/utils/init/miscinit.c	27 Feb 2003 18:46:39 -0000
> ***************
> *** 1044,1046 ****
> --- 1044,1135 ----
>   			 "which is not compatible with this version %s.",
>   			 file_major, file_minor, version_string);
>   }
> + 
> + /*-------------------------------------------------------------------------
> +  *				Library preload support
> +  *-------------------------------------------------------------------------
> +  */
> + 
> + #if defined(__mc68000__) && defined(__ELF__)
> + typedef int32 ((*func_ptr) ());
> + #else
> + typedef char *((*func_ptr) ());
> + #endif
> + 
> + /*
> +  * process any libraries that should be preloaded and
> +  * optionally pre-initialized
> +  */
> + void
> + process_preload_libraries(char *preload_libraries_string)
> + {
> + 	char	   *rawstring;
> + 	List	   *elemlist;
> + 	List	   *l;
> + 
> + 	if (preload_libraries_string == NULL)
> + 		return;
> + 
> + 	/* Need a modifiable copy of string */
> + 	rawstring = pstrdup(preload_libraries_string);
> + 
> + 	/* Parse string into list of identifiers */
> + 	if (!SplitIdentifierString(rawstring, ',', &elemlist))
> + 	{
> + 		/* syntax error in list */
> + 		pfree(rawstring);
> + 		freeList(elemlist);
> + 		elog(LOG, "invalid list syntax for preload_libraries configuration option");
> + 	}
> + 
> + 	foreach(l, elemlist)
> + 	{
> + 		char	   *tok = (char *) lfirst(l);
> + 		char	   *sep = strstr(tok, ":");
> + 		char	   *filename = NULL;
> + 		char	   *funcname = NULL;
> + 		func_ptr	initfunc;
> + 
> + 		if (sep)
> + 		{
> + 			/*
> + 			 * a colon separator implies there is an initialization function
> + 			 * that we need to run in addition to loading the library
> + 			 */
> + 			size_t		filename_len = sep - tok;
> + 			size_t		funcname_len = strlen(tok) - filename_len - 1;
> + 
> + 			filename = (char *) palloc(filename_len + 1);
> + 			memset(filename, '\0', filename_len + 1);
> + 			snprintf(filename, filename_len + 1, "%s", tok);
> + 
> + 			funcname = (char *) palloc(funcname_len + 1);
> + 			memset(funcname, '\0', funcname_len + 1);
> + 			snprintf(funcname, funcname_len + 1, "%s", sep + 1);
> + 		}
> + 		else
> + 		{
> + 			/*
> + 			 * no separator -- just load the library
> + 			 */
> + 			filename = pstrdup(tok);
> + 			funcname = NULL;
> + 		}
> + 
> + 		initfunc = (func_ptr) load_external_function(filename, funcname, false, NULL);
> + 		if (initfunc)
> + 			(*initfunc)();
> + 
> + 		elog(LOG, "preloaded library %s with initialization function %s", filename, funcname);
> + 
> + 		if (filename != NULL)
> + 			pfree(filename);
> + 
> + 		if (funcname != NULL)
> + 			pfree(funcname);
> + 	}
> + 
> + 	pfree(rawstring);
> + 	freeList(elemlist);
> + }
> + 
> Index: src/backend/utils/misc/guc.c
> ===================================================================
> RCS file: /opt/src/cvs/pgsql-server/src/backend/utils/misc/guc.c,v
> retrieving revision 1.115
> diff -c -r1.115 guc.c
> *** src/backend/utils/misc/guc.c	23 Feb 2003 23:27:21 -0000	1.115
> --- src/backend/utils/misc/guc.c	27 Feb 2003 18:46:39 -0000
> ***************
> *** 60,65 ****
> --- 60,66 ----
>   extern bool autocommit;
>   extern int	CommitDelay;
>   extern int	CommitSiblings;
> + extern char *preload_libraries_string;
>   
>   #ifdef HAVE_SYSLOG
>   extern char *Syslog_facility;
> ***************
> *** 812,817 ****
> --- 813,824 ----
>   	{
>   		{"lc_time", PGC_USERSET}, &locale_time,
>   		"C", locale_time_assign, NULL
> + 	},
> + 
> + 	{
> + 		{"preload_libraries", PGC_POSTMASTER, GUC_LIST_INPUT | GUC_LIST_QUOTE},
> + 		&preload_libraries_string,
> + 		"", NULL, NULL
>   	},
>   
>   	{
> Index: src/backend/utils/misc/postgresql.conf.sample
> ===================================================================
> RCS file: /opt/src/cvs/pgsql-server/src/backend/utils/misc/postgresql.conf.sample,v
> retrieving revision 1.70
> diff -c -r1.70 postgresql.conf.sample
> *** src/backend/utils/misc/postgresql.conf.sample	6 Feb 2003 20:25:33 -0000	1.70
> --- src/backend/utils/misc/postgresql.conf.sample	27 Feb 2003 18:46:39 -0000
> ***************
> *** 213,216 ****
> --- 213,217 ----
>   #transform_null_equals = false
>   #statement_timeout = 0		# 0 is disabled, in milliseconds
>   #db_user_namespace = false
> + #preload_libraries = ''
>    
> Index: src/include/miscadmin.h
> ===================================================================
> RCS file: /opt/src/cvs/pgsql-server/src/include/miscadmin.h,v
> retrieving revision 1.116
> diff -c -r1.116 miscadmin.h
> *** src/include/miscadmin.h	22 Feb 2003 05:57:45 -0000	1.116
> --- src/include/miscadmin.h	27 Feb 2003 18:46:39 -0000
> ***************
> *** 288,293 ****
> --- 288,294 ----
>   							 unsigned long id2);
>   
>   extern void ValidatePgVersion(const char *path);
> + extern void process_preload_libraries(char *preload_libraries_string);
>   
>   /* these externs do not belong here... */
>   extern void IgnoreSystemIndexes(bool mode);

> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/users-lounge/docs/faq.html

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman(at)candle(dot)pha(dot)pa(dot)us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

In response to

pgsql-hackers by date

Next:From: Bruce MomjianDate: 2003-03-17 17:32:30
Subject: Re: pg_dump / restore of empty database gives errors
Previous:From: TaralDate: 2003-03-17 17:23:47
Subject: Re: No index maximum? (was Re: No merge sort?)

pgsql-patches by date

Next:From: Bruce MomjianDate: 2003-03-17 17:30:12
Subject: Re: minor cleanup of regress.c
Previous:From: Bruce MomjianDate: 2003-03-17 17:21:53
Subject: Re: Minor doc patch: create function

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group