Skip site navigation (1) Skip section navigation (2)

Re: Allow substitute allocators for PGresult.

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)oss(dot)ntt(dot)co(dot)jp>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Allow substitute allocators for PGresult.
Date: 2011-12-01 10:48:03
Message-ID: 20111201.194803.07040624.horiguchi.kyotaro@oss.ntt.co.jp (view raw or flat)
Thread:
Lists: pgsql-hackers
Ouch! I'm sorry for making a reverse patch for the first modification.

This is an amendment of the message below. The body text is
copied into this message.

http://archives.postgresql.org/message-id/20111201.192419.103527179.horiguchi.kyotaro@oss.ntt.co.jp

=======
Hello, This is the next version of Allow substitute allocators
for PGresult.

Totally chaning the concept from the previous one, this patch
allows libpq to handle alternative tuple store for received
tuples.

Design guidelines are shown below.

 - No need to modify existing client code of libpq.

 - Existing libpq client runs with roughly same performance, and
   dblink with modification runs faster to some extent and
   requires less memory.

I have measured roughly of run time and memory requirement for
three configurations on CentOS6 on Vbox with 2GB mem 4 cores
running on Win7-Corei7, transferring (30 bytes * 2 cols) *
2000000 tuples (120MB net) within this virutal machine. The
results are below.

		    	    	  xfer time	Peak RSS
Original	    	    	: 6.02s		850MB
libpq patch + Original dblink	: 6.11s		850MB
full patch    	       		: 4.44s		643MB

xfer time here is the mean of five 'real time's measured by
running sql script like this after warmup run.

=== test.sql
select dblink_connect('c', 'host=localhost port=5432 dbname=test');
select * from dblink('c', 'select a,c from foo limit 2000000') as (a text, b bytea) limit 1;

select dblink_disconnect('c');
===
$  for i in $(seq 1 10); do time psql test -f t.sql; done 
===

Peak RSS is measured by picking up heap Rss in /proc/[pid]/smaps.


It seems somewhat slow using patched libpq and original dblink,
but it seems within error range too. If this amount of slowdown
is not permissible, it might be improved by restoring the static
call route before for extra redundancy of the code.

On the other hand, full patch version seems obviously fast and
requires less memory. Isn't it nice?

This patch consists of two sub patches.

The first is a patch for libpq to allow rewiring tuple storage
mechanism. But default behavior is not changed. Existing libpq
client should run with it.

The second is modify dblink to storing received tuples into
tuplestore directly using the mechanism above.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



Attachment: dblink_direct_tuplestore_v1.patch
Description: text/x-patch (11.4 KB)
Attachment: libpq_subst_storage_v1.patch
Description: text/x-patch (15.8 KB)

In response to

Responses

pgsql-hackers by date

Next:From: Greg SmithDate: 2011-12-01 10:56:36
Subject: Re: [PATCH] optional cleaning queries stored in pg_stat_statements
Previous:From: Kyotaro HORIGUCHIDate: 2011-12-01 10:24:19
Subject: Re: Allow substitute allocators for PGresult.

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group