Re: Rewriting the test of pg_upgrade as a TAP test - take two

On Fri, Mar 2, 2018 at 4:38 PM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> Hi,
>
> On 2018-01-26 17:00:26 +0900, Michael Paquier wrote:
>> Another topic that I would like to discuss is how this interface is fit
>> for the buildfarm code. After hacking my stuff, I have looked at the
>> buildfarm code to notice that things like TestUpgradeXversion.pm do
>> *not* make use of test.sh, and that what I hacked is much similar to
>> what the buildfarm code is doing, which is itself roughly a copy of what
>> test.sh does. Andrew, your feedback would be important here.
>
> Andrew, any comments?
>
>

I'll take a look.

Meanwhile:

TestUpgradeXversion.pm is different in a number of respects from the
inbuilt test regime. It doesn't run the normal regression suite to set
up a test. Rather, it saves out the installed binaries and data
directory of the buildfarm client, and then tries to upgrade those
saved data directorories from earlier branches or the current branch
to the target version being tested. It has to make a few adjustments
to the databases along the way. It does a compare of pre- and post-
pg_dumpall runs, allowing a fuzz factor if the versions are different.
The other upside to the scheme is that we're testing pg_dump against
earlier branches as part of testing pg_upgrade.

All this consumes quite a bit of disk space - currently 3.5Gb between
runs on crake. That could probably be reduced some by removing log as
we go.

I don't think a scheme like this is going to be terribly workable
outside some system such as the buildfarm that deals with multiple
branches.

One of the significant pluses to TestUpgradeXversion.pm is that it
tests upgrading quite a bit more than the standard regression
database. It also tests all the contrib databases and the isolation
and pl_regression databases. Several bugs have been found that way,
IIRC, and we should arguably do something along the same lines for our
builtin testing.

I'll post a follow up when I've had a chance to have a good look at
what Michael has actually done.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services