Skip site navigation (1) Skip section navigation (2)

64-bit pgbench V2

From: Greg Smith <greg(at)2ndquadrant(dot)com>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: 64-bit pgbench V2
Date: 2010-07-05 23:48:22
Message-ID: 4C326F46.4050801@2ndquadrant.com (view raw or flat)
Thread:
Lists: pgsql-hackers
Attached is an updated second rev of the patch I sent a few months ago, 
to expand pgbench to support database scales larger than around 
4,294--where the 32-bit integer for the account number overflows in the 
current version.  The current limit makes for about a 60GB database.  
Last week I ran this on a system with 72GB of RAM, which are already 
quite common, and wasn't able to get a test that didn't fit in RAM.  
Without a bug fix here I am concerned that pgbench will ship in 9.0 
already obsolete for the generation of hardware is it going to be 
deployed on.

The main tricky part was figuring how to convert the \setshell 
implementation.  That uses strtol to parse the number that should have 
been returned by the shell call.  It turns out there are a stack of ways 
to do something similar but return 64 bits instead:

* strtoll is defined by ISO C99
* strtoq was used on some earlier BSD systems
* MSVC has _strtoi64 for signed and _strtoui64 for unsigned 64bit integers

According to the glib docs at 
http://www.gnu.org/software/gnulib/manual/html_node/strtoll.html , 
stroll is missing on HP-UX 11, OSF/1 5.1, Interix 3.5, so one of the 
HP-UX boxes might be a useful testbed for what works on a trickier platform.

For prototype purposes, I wrote the patch to include some minimal logic 
to map the facility available to strtoint64, falling back to the 32-bit 
strtol if that's the best available.  There are three ways I could 
forsee this going:

1) Keep this ugly bit of code isolated to pgbench
2) Move it to src/include/c.h where the other 64-bit int abstraction is done
3) Push the problem toward autoconf

I don't have a clear argument for or against those individual options, 
they all seem reasonable from some perspectives.

The only open issue I'm not sure about is whether the situation where 
the code falls back to 32-bits should be documented, or even a warning 
produced if you create something at a scale without some strtoll 
available.  Given that it only impacts the \setrandom case, it's not 
really a disaster that it might not work, so long as there's 
documentation explaining the potential limitations.  I'll write those if 
necessary, but I think that some testing on known tricky platforms that 
I don't have setup here is the best next step, so I'm looking for 
feedback on that.

-- 
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com   www.2ndQuadrant.us


Attachment: pgbench-64-v2.patch
Description: text/x-patch (4.9 KB)

Responses

pgsql-hackers by date

Next:From: Tom LaneDate: 2010-07-06 00:17:24
Subject: Re: 64-bit pgbench V2
Previous:From: Robert HaasDate: 2010-07-05 21:57:37
Subject: Re: pessimal trivial-update performance

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group