Skip site navigation (1) Skip section navigation (2)

Re: Help with bulk read performance

From: Dan Schaffer <Daniel(dot)S(dot)Schaffer(at)noaa(dot)gov>
To: Andy Colson <andy(at)squeakycode(dot)net>
Cc: Jim Nasby <jim(at)nasby(dot)net>, pgsql-performance(at)postgresql(dot)org, Nick Matheson <Nick(dot)D(dot)Matheson(at)noaa(dot)gov>
Subject: Re: Help with bulk read performance
Date: 2010-12-15 20:15:14
Message-ID: (view raw, whole thread or download thread mbox)
Lists: pgsql-performance
My name is Dan and I'm a co-worker of Nick Matheson who initially submitted this question (because the mail group had me blacklisted 
for awhile for some reason).

Thank you for all of the suggestions.  We were able to improve out bulk read performance from 3 MB/s to 60 MB/s (assuming the data 
are NOT in cache in both cases) by doing the following:

1. Storing the data in a "bytea" column instead of an "array" column.
2. Retrieving the data via the Postgres 9 CopyManager#copyOut(String sql, OutputStream stream)  method

The key to the dramatic improvement appears to be the reduction in packing and unpacking time on the server and client, 
respectively.  The server packing occurs when the retrieved data are packed into a bytestream for sending across the network. 
Storing the data as a simple byte array reduces this time substantially.  The client-side unpacking time is spent generating a 
ResultSet object. By unpacking the bytestream into the desired arrays of floats by hand instead, this time became close to negligible.

The only downside of storing the data in byte arrays is the loss of transparency.  That is, a simple "select *" of a few rows shows 
bytes instead of floats.  We hope to mitigate this by writing a simple stored procedures that unpacks the bytes into floats.

A couple of other results:

If the data are stored as a byte array but retrieve into a ResultSet, the unpacking time goes up by an order of magnitude and the 
observed total throughput is 25 MB/s.  If the data are stored in a Postgres float array and unpacked into a byte stream, the 
observed throughput is 20 MB/s.

Dan (and Nick)

Andy Colson wrote:
> On 12/14/2010 9:41 AM, Jim Nasby wrote:
>> On Dec 14, 2010, at 9:27 AM, Andy Colson wrote:
>>> Is this the same thing Nick is working on?  How'd he get along?
>> So it is. The one I replied to stood out because no one had replied to 
>> it; I didn't see the earlier email.
>> -- 
>> Jim C. Nasby, Database Architect                   jim(at)nasby(dot)net
>> 512.569.9461 (cell)               
> Oh.. I didn't even notice the date... I thought it was a new post.
> But still... (and I'll cc Nick on this)  I'd love to hear an update on 
> how this worked out.
> Did you get it to go fast?  What'd you use?  Did the project go over 
> budget and did you all get fired?  COME ON MAN!  We need to know! :-)
> -Andy

Attachment: daniel_s_schaffer.vcf
Description: text/x-vcard (252 bytes)

In response to


pgsql-performance by date

Next:From: Divakar SinghDate: 2010-12-16 08:10:39
Subject: Re: performance libpq vs JDBC
Previous:From: Werner ScholtesDate: 2010-12-15 14:51:55
Subject: performance libpq vs JDBC

Privacy Policy | About PostgreSQL
Copyright © 1996-2017 The PostgreSQL Global Development Group