Re: Python/pytest test framework take two

Hi,

On 2026-08-19 10:59:16 +0200, Jelte Fennema-Nio wrote:
> Based on various discussions there seems to be enough desire for writing
> tests using pytest. The previous mailinglist discussion[1] became quite
> long and discussed various different aspects. So I'm starting a new
> thread with my most recent attempt at a "nice to use" framework. It has
> many changes since the last one.
>
> The changes resulted from my experience converting ~half of the Perl
> tests[2] using an LLM, which revealed gaps and awkward usage patterns.
> The last couple of patches in this patchset are a few of those
> conversions. These are NOT meant to be committed, at least not until
> this thread[3] reaches a conclusion about conversions. They are only
> meant to showcase the framework for now.
>
> The patches in the patchset have detailed commit messages. But here's a
> short summary per patch:
> 0001: Small patch to make meson CI output better in general by not
> truncating error logs as much

-1. I don't see why we would want to do that. Printing 10k lines onto the fake
terminal in CI makes no sense whatsoever. If the pytest errors are unreadably
verbose, we need to fix that, not display more unusable output.

> 0002: Small meson refactoring, moving a magic number into a variable

WFM.

> 0003: Installing 32-bit python, can be removed after pg-vm-images PR is
> merged

-1. As commented on that PR just now, this is a maintenance burden going
forward that likely Bilal or I will have to pay.

> 0004: Basic pytest support for the build system and CI.
> 0005: Introduction of the actual framework that can manage Postgres
> servers and send queries

Phew, that's quite a bit of code to review :(. Not sure there's much that can
be done about that though...

> 0006-0007: Two short converted tests that run much faster using this
> framework
> 0008: Shows off the background_sql functionality, which is much easier
> to use and more robust than Perl its background_psql

> 0009: Conversion of the 001_aio test, which is the most unreadable Perl
> test I could find (sorry Andres). It becomes, less buggy, more
> readable (imo), shorter and faster when written in Python.

No reason to be sorry, I hated writing it.

> 0010-0012: Some more tests that are faster and/or easier to read+write
> in Python.

What's your plan for validating that these do not actually reduce test
coverage substantially?

I think this needs some systematic work. E.g. build with coverage, clear all
coverage, running the perl test, collect coverage, clear coverage, run the
python test, compare the two coverages, including branch coverage.

Greetings,

Andres Freund