Re: pg_upgrade automatic testing

On fre, 2011-09-02 at 15:04 -0400, Tom Lane wrote:
> IMO there's next to no value in testing that scenario anyway, since
> nobody would ever use it in the field. What *would* be of value is
> testing upgrades from previous release versions. Probably that will
> take some work in the buildfarm infrastructure as well as figuring out a
> non-problematic test case to use, but that's the direction we need to
> head in.

Well, let's take a step back. I originally developed this test runner a
few months ago while fixing upgrade issues related to composite types
that had been altered. So this is actually useful stuff that would help
prevent these sorts of problems in the future, and would help developers
fix problems of this sort.

But if you think about it, it doesn't really test pg_upgrade, it tests
pg_dump. So the test could just as well be moved to src/bin/pg_dump/
and be labeled "pg_dump smoke test" or whatever. (Minor detail: The bug
fix above involved the --binary-upgrade flag, so it is somewhat
pg_upgrade related.)

A real pg_upgrade test suite should naturally upgrade across binary
incompatible versions. The real question is how you develop a useful
test input. Most pg_upgrade issues are not bugs of omission or
regression but unexpected corner cases discovered with databases of
nontrivial usage patterns. (The recent one related to upgrade from 8.3
is an exception.) Because the basic premise of pg_upgrade is, dump and
restore the schema, move over the files, that's it, and the rest of the
code is workarounds for obscure details that are difficult to anticipate
let alone test for.