Re: Direction for test frameworks: Perl TAP vs. Python/pytest

On 2026-06-17 We 12:01 PM, Jacob Champion wrote:
> On Wed, Jun 17, 2026 at 6:39 AM Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> OK, so the consensus I'm sensing is that we should add a pytest
>> framework, backpatch it into all the live branches, and initially port
>> some relatively contained set of tests and backpatch that too.
> As for "new tests that can't easily be written today": I've already
> written several protocol-level tests for OAuth that implement broken
> client and server behavior (they're in the pg-pytest-suite thread).
> Those could be ported for review on top of the new framework. I could
> look at some of the more recent ad-hoc Perl socket tests at the same
> time.
>
>> If I had
>> to pick a candidate suite it would be the recovery tests (and my patch
>> set conveniently contains a patch specifically for those).
> No strong opinions here on the candidates for initial backport -- but
> I will note that src/test/recovery doesn't look like a small slice, at
> 13k lines and more than half of the Test::Cluster API used.
>
>> I guess
>> that means the Test::Session work gets left on the cutting room floor -
>> if we run into issues with perl tests the answer will be to convert them
>> to pytest.
> If you think Test::Session is good to go, I see no reason not to
> improve the Perl tests (since there seems to be a growing consensus
> that the conversion won't happen overnight).

There's a fairly high amount of friction here, though. If we didn't plan
to replace the perl tests it would be worth it, but I'm not sure it's
worth it if we are. To keep things in sync, which people have expressed
concerns about, we'd need to backport it. The whole set is not humungous
but it's not tiny either:

59 files changed, 3317 insertions(+), 1113 deletions(-)

Then there's the matter of needing either libffi + FFI::Platypus or
needing to create/maintain an XS wrapper.

With pytest I feel much more confident about backpatching it because all
we need is python itself + pytest and ideally pexpect;

So I'm prepared to put the work in if people think it's a good idea, but
I want a fairly solid indication of that.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com