Direction for test frameworks: Perl TAP vs. Python/pytest

Hi,

We now have several pieces of test-infrastructure work in flight that
point in
different directions, and I'd like the community to settle the direction
before
more effort is expended on directions we might not want to follow.

Several things exist today:

1. My PostgreSQL::Test::Session work, which gives the Perl TAP framework
   in-process libpq sessions instead of forking psql per query. Part of the
   motivation is that BackgroundPsql is fragile and has a habit of
swallowing
   errors; a real session object sidesteps that. It also means far less
   forking, so things run a good deal more efficiently -- especially on
   Windows, where process creation is expensive. The Perl framework is
still
   gaining features. It currently relies on FFI::Platypus, and it's not
clear
   that's available everywhere we build Postgres -- if not, we'd need
to fall
   back on an XS wrapper. ("Pq Session object plus TAP test improvements",
   CF #6834 [1]; origin thread "Using LibPq in TAP tests via FFI" [2].)

2. Jelte Fennema-Nio's proposed minimal Python/pytest framework
   (src/test/pytest), aimed mainly at new tests. It already carries an
   in-process ctypes libpq layer of its own -- and note that on the Python
   side this needs no FFI module or XS wrapper at all, since ctypes is
in the
   standard library and can bind to libpq natively. ("RFC: adding
pytest as a
   supported test framework", CF #6045 [3].)

3. Two full ports of the existing TAP suite (~300 tests plus the
framework) to
   that pytest shape, both validated against the existing suite. Greg
Burd's [4]
   is a fully additive port: it leaves the existing Perl suite in place
and runs
   a Python twin of each test beside it. Queries go through psql by default
   (matching safe_psql), with an opt-in in-process ctypes libpq layer for
   protocol-level tests. Mine [5] builds on Jelte's framework rather than
   standing apart from it: I've modularised and extended its libpq
layer into a
   full Session along the lines of Test::Session, expanded the server and
   fixture helpers to cover replication, logical decoding, backups, and
log and
   connection/auth checks, and added harness support for the SSL, LDAP,
   Kerberos and OAuth suites along with a pg_regress runner. I did this
partly
   to show that we have an adequate basis for porting even if we never
declare
   a flag day, and partly because, for what it's worth, I think the
   Test::Session model is a much more solid foundation for a port than the
   current framework is.

So the work supports more than one strategy. The question is which we want:

(A) Wholesale port with a flag day -- Python becomes *the* framework,
migrate the existing tests, stop accepting and eventually remove the
Perl one.
One framework to learn and maintain, but a large disruptive change with real
dependency questions (which Python, which libpq binding, buildfarm
minimums).

(B) Minimal Python framework for new tests, Perl preserved alongside.
Less disruptive, but commits us to two frameworks for some time.

(C) Something in between -- adopt the Python framework and port over
just the
most fragile tests (the ones leaning on BackgroundPsql, say), leaving
the rest
in Perl for now. This gets the worst pain points onto a sounder footing
without committing to a full migration up front.

(D) Keep both, but put each on a sounder footing: land the Test::Session
work
so the Perl side stops depending on BackgroundPsql, and adopt the Python
framework alongside it for new tests. This addresses the fragility
within Perl
rather than by porting, and doesn't force a migration either way.

If we land on (B), (C) or (D), I think we have to answer these questions:

- Do we implement new infrastructure features (e.g. the Session work) twice,
  or let the two frameworks drift?
- When an existing Perl test needs substantial new coverage, do we extend it
  in Perl or port it? "Port on touch" front-loads cost onto bug fixers.

Having shown a full port is feasible, my inclination is that the maintenance
argument favours converging on one framework -- but I'm wary of the
disruption,
so I could see (D) as a sane and fairly safe option.

So how would people like to proceed?

cheers

andrew

[1] https://commitfest.postgresql.org/patch/6834/
[2]
https://www.postgresql.org/message-id/997f67ac-cc02-4076-9fda-db465e89c622%40dunslane.net
[3] https://commitfest.postgresql.org/patch/6045/
[4] https://github.com/gburd/postgres/pull/24
[5] https://github.com/adunstan/pgdev/pull/5

--
Andrew Dunstan
EDB: https://www.enterprisedb.com