Re: pgbench is badly broken since July

I'll look into this.
--
Tatsuo Ishii
SRA OSS, Inc. Japan

> I have just realized that the recent patches in pgbench have altered its
> behavior in a way that destroys reproducibility of results --- I'm
> seeing reported TPS numbers about twice what they were before that.
> I'd love to say we did something in the past month that made the backend
> 2X faster, but sadly, tain't so :-(
>
> I've only done limited investigation but what seems to be happening
> is that the "-s" scale factor is no longer being applied to adjust the
> range of the random variables, leading to only part of the database
> being touched. For example, with -s = 10, the older code updates all 10
> "branches" rows but CVS HEAD only seems to touch bid = 1.
>
> I see that the code still thinks it is scaling the numbers, but the
> mechanism for doing that has changed drastically, and I suspect that's
> the problem. I assume there's some garden-variety bug in there.
>
> At a more fundamental level, the recent patches seem to mean that
> pg_dump is doing noticeably more work to issue its queries than it was
> before, and I wonder whether this isn't going to destroy cross-version
> reproducibility in its own way. Have you done any timing studies to
> verify that the new and old code have comparable performance? I don't
> mind enhancing pgbench's functionality for non-default tests, but I
> think it's important that the default case remain comparable over time.
> If there is a noticeable speed difference then I'd vote for going back
> to the klugy old code for adjusting the default script's values.
>
> And at the nitpicking level, is ":tps" a good name for the variable that
> reflects the scaling factor? It seems awfully easy to confuse that with
> the TPS numbers that pgbench reports. Perhaps ":scale" or some such
> would be better.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>