Re: Help with bulk read performance

Hi,
My name is Dan and I'm a co-worker of Nick Matheson who initially submitted this question (because the mail group had me blacklisted
for awhile for some reason).

Thank you for all of the suggestions. We were able to improve out bulk read performance from 3 MB/s to 60 MB/s (assuming the data
are NOT in cache in both cases) by doing the following:

1. Storing the data in a "bytea" column instead of an "array" column.
2. Retrieving the data via the Postgres 9 CopyManager#copyOut(String sql, OutputStream stream) method

The key to the dramatic improvement appears to be the reduction in packing and unpacking time on the server and client,
respectively. The server packing occurs when the retrieved data are packed into a bytestream for sending across the network.
Storing the data as a simple byte array reduces this time substantially. The client-side unpacking time is spent generating a
ResultSet object. By unpacking the bytestream into the desired arrays of floats by hand instead, this time became close to negligible.

The only downside of storing the data in byte arrays is the loss of transparency. That is, a simple "select *" of a few rows shows
bytes instead of floats. We hope to mitigate this by writing a simple stored procedures that unpacks the bytes into floats.

A couple of other results:

If the data are stored as a byte array but retrieve into a ResultSet, the unpacking time goes up by an order of magnitude and the
observed total throughput is 25 MB/s. If the data are stored in a Postgres float array and unpacked into a byte stream, the
observed throughput is 20 MB/s.

Dan (and Nick)

Andy Colson wrote:
> On 12/14/2010 9:41 AM, Jim Nasby wrote:
>> On Dec 14, 2010, at 9:27 AM, Andy Colson wrote:
>>> Is this the same thing Nick is working on? How'd he get along?
>>>
>>> http://archives.postgresql.org/message-id/4CD1853F.2010800@noaa.gov
>>
>> So it is. The one I replied to stood out because no one had replied to
>> it; I didn't see the earlier email.
>> --
>> Jim C. Nasby, Database Architect jim(at)nasby(dot)net
>> 512.569.9461 (cell) http://jim.nasby.net
>>
>>
>>
>
> Oh.. I didn't even notice the date... I thought it was a new post.
>
> But still... (and I'll cc Nick on this) I'd love to hear an update on
> how this worked out.
>
> Did you get it to go fast? What'd you use? Did the project go over
> budget and did you all get fired? COME ON MAN! We need to know! :-)
>
> -Andy
>