Skip site navigation (1) Skip section navigation (2)

Re: Space management for PGresult

From: Simon Riggs <simon(at)2ndquadrant(dot)com>
To: Atsushi Ogawa <atsushi(dot)ogawa(at)gmail(dot)com>
Cc: pgsql-patches(at)postgresql(dot)org
Subject: Re: Space management for PGresult
Date: 2005-11-24 17:22:00
Message-ID: 1132852920.4347.92.camel@localhost.localdomain (view raw or flat)
Thread:
Lists: pgsql-patches
On Wed, 2005-11-23 at 16:21 +0900, Atsushi Ogawa wrote:
> In space management for PGresult of libpq, the block size of PGresult
> is always PGRESULT_DATA_BLOCKSIZE(2048bytes). Therefore, when a large
> result of query is received, malloc is executed many times.
> 
> My proposal is to enlarge the size of the block whenever the block is
> allocated. The size of first block is PGRESULT_DATA_BLOCKSIZE. And the
> size of the following blocks will be doubled until it reaches
> PGRESULT_MAX_DATA_BLOCKSIZE.
> 
> PGRESULT_MAX_DATA_BLOCKSIZE is new constants. I think that 2Mbytes is
> good enough for this constants.
> 
> The test result is the following:
> 
> Test SQL:
> select * from accounts; (It is pgbench's table. scale factor is 10.)
> 
> The number of malloc calls at pqResultAlloc:
>  8.1.0  : 80542
>  patched:    86
> 
> Execution time:
>  8.1.0  : 6.80 sec
>  patched: 6.73 sec
> 

What this highlights for me is that we have (IMHO) a strange viewpoint
on allocating result memory, not an optimization issue. 

We really ought to be streaming the result back to the user, not
downloading it all into a massive client side chunk of memory. It ought
to be possible to do this with very low memory, and would probably have
the side-effect of reducing time-to-first-row. Then we wouldn't have a
memory allocation issue at all.

Consider what will happen if you do "select * from too_big_table". We'll
just run for ages, then blow memory and fail. (That's what it used to
do, does it still? he asks lazily).

Best Regards, Simon Riggs


In response to

Responses

pgsql-patches by date

Next:From: Tom LaneDate: 2005-11-24 17:32:01
Subject: Re: Space management for PGresult
Previous:From: Tom LaneDate: 2005-11-24 14:55:24
Subject: Re: Space management for PGresult

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group