Re: pgbench --unlogged-tables

On 07/25/2011 09:23 AM, Robert Haas wrote:
> At some point, we also need to sort out the scale factor limit issues,
> so you can make these things bigger.
>

I had a patch to improve that whole situation, but it hasn't seem to nag
at me recently. I forget why it seemed less important, but I doubt I'll
make it another six months without coming to some resolution there.

The two systems I have in for benchmarking right now have 128GB and
192GB of RAM in them, so large scales should have been tested.
Unfortunately, it looks like the real-world limiting factor on doing
lots of tests at big scales is how long it takes to populate the data
set. For example, here's pgbench creation time on a big server (48
cores, 128GB RAM) with a RAID10 array, when scale=20000 (292GB):

real 174m12.055s
user 17m35.994s
sys 0m52.358s

And here's the same server putting the default tablespace (but not the
WAL) on [much faster flash device I can't talk about yet]:

Creating new pgbench tables, scale=20000
real 169m59.541s
user 18m19.527s
sys 0m52.833s

I was hoping for a bigger drop here; maybe I needed to use unlogged
tables? (ha!) I think I need to start looking at the pgbench data
generation stage as its own optimization problem. Given how expensive
systems this large are, I never get them for very long before they are
rushed into production. People don't like hearing that just generating
the data set for a useful test is going to take 3 hours; that tends to
limit how many of them I can schedule running.

And, yes, I'm going to try and sneak in some time to test fastpatch
locking on one of these before they head into production.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us