Re: Direction for test frameworks: Perl TAP vs. Python/pytest

On Wed, Jun 24, 2026 at 8:39 AM Peter Eisentraut <peter(at)eisentraut(dot)org>
wrote:

> On 16.06.26 17:23, Daniel Gustafsson wrote:
> >> That's a fair concern. It seems to me that back-patching TAP tests
> >> doesn't happen too often.
> > A quick look at REL_14 shows that there have been 450+ commits touching
> > src/test in some fashion since 14.0 was stamped. It's not the same as
> > backpatching of TAP tests but it's a data point.
>
> git log --oneline REL_14_0..REL_14_STABLE -- **/t/*.pl src/test/perl | wc
> -l
> 172
>
> So a bit less than 450.
>
> So basically what you'd sign up for is that about 35 test changes per
> year have to be written twice.
>
> I mean, even in the current setup, there are sometimes significant
> differences in the Perl test framework between major versions. So the
> current number isn't zero.
>
> Personally, I find the idea of doing a wholesale conversion of the tests
> in a backbranch to be horrible. I wouldn't trust such a thing. And the
> amount of work to actually put that together in the first place would
> most likely be much larger than converting test changes on the go as
> needed.
>

I want to chime in that I am interested in supporting this effort. My
company has decided to sponsor Salma (from the BTree Bloat project) in
assisting with this effort, starting late 2026 and definitely early 2027
for at least a year, possibly two.

But I want to make sure we can work together, I don't want to redo existing
frameworks or duplicate effort.
I am confused about "not trusting the backpatched tests". (I was
envisioning we back-port the tests once working in master, if acceptable)

Also, I want to confirm we are all talking about the same process to get
there. I think we can clearly define success as having replaced all of the
perl tests with python tests, and no longer running perl tests.

Along the way, I had envisioned running BOTH sets of tests until that
happened. This way, if something were caught by one and not the other, it
would stick out. (Now this is an incomplete thought, but it made sense to
me). Without that, I would definitely say we must literally validate that
the python tests catch the errors that the perl tests would catch (t vs.
True, etc). Otherwise, how do you trust the new tests?

Once that is agreed upon (whatever the answer), the next piece would be
about picking an area (contrib?) and rewriting those tests, one module at a
time, and routing them amongst this group for deep feedback, proper
approaches, etc. Before we then start pushing.

Also, our intention is to have AI (Multiple Agents), create the Python
tests from the perl code, and other agents validate the coverage, etc. And
humans (Salma, myself, others interested) review. Before Salma pushes a
patch up for review. (Hence the early feedback, and moving more slowly at
first). Maybe even developing some "proper approaches" so we have fewer
"timing" issues...

Again, we want to be 100% transparent about this approach up front. We
would also be willing to share the configuration and setup, so if other
contributors wanted to replicate the process or review and validate it
differently... They could.

Kirk
PS: Even if we got this all done in 2027, it would be like 2031 before all
the perl tests would be gone, unless we backport a couple of versions.
Again, not sure where the group is at on those nuances.