Re: Direction for test frameworks: Perl TAP vs. Python/pytest

On 2026-06-16 Tu 7:02 PM, Michael Paquier wrote:
> On Tue, Jun 16, 2026 at 12:20:12PM -0400, Tom Lane wrote:
>> Jacob Champion <jacob(dot)champion(at)enterprisedb(dot)com> writes:
>>> It would probably be pretty easy to do that in slices. Backport the
>>> Python core, then develop the replacement for a segment of related
>>> tests as a reviewable and backportable chunk. That lets us refactor
>>> the testing APIs as we go, and if there are failures after backport,
>>> at least it's just that one area of the tests and not "the whole
>>> thing".
>> Makes sense.
> Backporting the core Python infrastructure sounds like a first
> mandatory step here. Under time-pressure with a release timeline,
> complicated/sensitive fixes may require tests across all the stable
> branches. Not having to translate the same test across two
> infrastructures would save a lot of brain resources.
>
> Backporting changes of the tests into the new Python infrastructure
> would make backpatching less complicated, indeed. It is much more
> common for me to add tests to existing scripts than create new ones.
> For a routine fix, I'd be OK to switch an existing test from perl to
> Python before doing a backpatch and expanding a script. Now, I don't
> think that we should close the door to the possibility of having to
> add some perl code in an existing perl script if we are under a time
> constraint (imagine in the security area).
>
> The only thing I'd argue to avoid is having to maintain two versions
> of the same test across *two* infrastructures.

OK, so the consensus I'm sensing is that we should add a pytest
framework, backpatch it into all the live branches, and initially port
some relatively contained set of tests and backpatch that too. If I had
to pick a candidate suite it would be the recovery tests (and my patch
set conveniently contains a patch specifically for those). It's the
suite that has had the most churn, and has also been somewhat fragile
historically. Then we can convert tests in manageable chunks. I guess
that means the Test::Session work gets left on the cutting room floor -
if we run into issues with perl tests the answer will be to convert them
to pytest.

This is going to make Python a hard requirement for non-psql based
testing. There are currently 7 buildfarm animals building master without
enabling python, although 4 of those are building with meson so
presumably python is available to them.

It's also going to mean that some of us (including me) will need to
vastly improve our python fluency.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com