Re: Test to dump and restore objects left behind by regression

On 22.02.24 02:01, Michael Paquier wrote:
> On Wed, Feb 21, 2024 at 12:18:45PM +0530, Ashutosh Bapat wrote:
>> Even with 1 and 2 the test is useful to detect dump/restore anomalies.
>> I think we should improve 3, but I don't have a good and simpler
>> solution. I didn't find any way to compare two given clusters in our
>> TAP test framework. Building it will be a lot of work. Not sure if
>> it's worth it.
>
> + my $rc =
> + system($ENV{PG_REGRESS}
> + . " $extra_opts "
> + . "--dlpath=\"$dlpath\" "
> + . "--bindir= "
> + . "--host="
> + . $node->host . " "
> + . "--port="
> + . $node->port . " "
> + . "--schedule=$srcdir/src/test/regress/parallel_schedule "
> + . "--max-concurrent-tests=20 "
> + . "--inputdir=\"$inputdir\" "
> + . "--outputdir=\"$outputdir\"");
>
> I am not sure that it is a good idea to add a full regression test
> cycle while we have already 027_stream_regress.pl that would be enough
> to test some dump scenarios. These are very expensive and easy to
> notice even with a high level of parallelization of the tests.

The problem is, we don't really have any end-to-end coverage of

dump
restore
dump again
compare the two dumps

with a database with lots of interesting objects in it.

Note that each of these steps could fail.

We have somewhat relied on the pg_upgrade test to provide this testing,
but we have recently discovered that the dumps in binary-upgrade mode
are different enough to not test the normal dumps well.

Yes, this test is a bit expensive. We could save some time by doing the
first dump at the end of the normal regress test and have the pg_dump
test reuse that, but then that would make the regress test run a bit
longer. Is that a better tradeoff?

I have done some timing tests:

master:

pg_dump check: 22s
pg_dump check -j8: 8s
check-world -j8: 2min44s

patched:

pg_dump check: 34s
pg_dump check -j8: 13s
check-world -j8: 2min46s

So overall it doesn't seem that bad.