Re: [9.3 CF 1] 2 Weeks In & The problem with Performance Patches

On Fri, Jun 28, 2013 at 4:16 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> (2) ideas on how we can speed up/parallelize performance testing efforts
> are extremely welcome.

An official perf-test script in GIT, even if it only tests general
pg-bench-like performance, that can take two builds and compare them,
like unladen-swallow's perf.py[0][1], would enable regular folk
perform the testing on various hardware configurations and contribute
their results more easily. Results would be in standardized format,
which would help on the interpretation of those numbers, and patches
would also be able to add patch-specific tests to the script's
battery.

I had a bash script that ran a few tests I used when evaluating the
readahead patch. It's very very green, and very very obvious, so I
doubt it'd be useful, but if noone else has one, I could dig through
my backups.

The test handled the odd stuff about clearing the OS cache between
test runs, and stuff like that, which is required for meaningful
results. I think this is where an official perf test script can do
wonders: accumulate and share knowledge on testing methodology.

[0] http://code.google.com/p/unladen-swallow/wiki/Benchmarks
[1] http://code.google.com/p/unladen-swallow/source/browse/tests/perf.py