Re: Python/pytest test framework take two

On Wed, 19 Aug 2026 at 15:16, Andres Freund <andres(at)anarazel(dot)de> wrote:
> -1. I don't see why we would want to do that. Printing 10k lines onto the fake
> terminal in CI makes no sense whatsoever. If the pytest errors are unreadably
> verbose, we need to fix that, not display more unusable output.

I agree that 10k is way too much, and also not necessary. Some of the
failed output has been very verbose though, and I have some ideas on
how to fix that (silence some commands, limit size of logs shown in
the output, etc). I'll work on that for the next version. However, the
Meson default of 100 lines is very small. I think we should bump that
to a few hundred e.g. 300-500 lines. There's quite a bit of *useful*
output that we can show that still easily overflows a 100 line limit:
1. Pytest shows the stack trace of an error
2. Values of local variables
3. Multiline diff view of the diff between two lists if an asserted
comparison on them fails
4. stderr, stdout and python logging
5. PostgreSQL logs of the servers

> > 0003: Installing 32-bit python, can be removed after pg-vm-images PR is
> > merged
>
> -1. As commented on that PR just now, this is a maintenance burden going
> forward that likely Bilal or I will have to pay.

I responded in that PR, let's discuss there further. I'm fine using
another approach, but we need a solution for 32bit.

> > 0004: Basic pytest support for the build system and CI.
> > 0005: Introduction of the actual framework that can manage Postgres
> > servers and send queries
>
> Phew, that's quite a bit of code to review :(. Not sure there's much that can
> be done about that though...

Yeah, I'm also not sure how to make it much better. I guess some
methods on the PostgresServer could be left out of the initial commit
and added in a follow-up one. But I don't think that would improve the
situation much in the end.

> No reason to be sorry, I hated writing it.

I can only imagine...

> > 0010-0012: Some more tests that are faster and/or easier to read+write
> > in Python.
>
> What's your plan for validating that these do not actually reduce test
> coverage substantially?
>
> I think this needs some systematic work. E.g. build with coverage, clear all
> coverage, running the perl test, collect coverage, clear coverage, run the
> python test, compare the two coverages, including branch coverage.

To be clear on my intent with this patchset: I DO NOT want to make
converting all/most/some perl tests a goal of this patchset. The ones
converted here are purely intended as a review-aid to show the pytest
framework in action. i.e. so reviewers can see what a test would look
like, as well as serving as a proof that it supports the functionality
we need to write a wide range of tests. Merging support for writing
Pytest tests would be a huge win on its own, even without converting
any tests. Tests for new features could then be written using it. And
more practically speaking: I don't want to spend my energy discussing
the details and difficulties of converting tests until we have a
pytest framework in main.

But responding to your thoughts on it anyway: I agree that if/when we
start converting Perl tests to Python, we should have do coverage
measurements