Re: Python/pytest test framework take two

Hi

I came here from the pg_stat_progress_vacuum thread [1], where Michael
pointed to Python for tests like that.

On 2026-08-19, Andres Freund wrote:
> What's your plan for validating that these do not actually reduce test
> coverage substantially?
>
> I think this needs some systematic work. E.g. build with coverage, clear
all
> coverage, running the perl test, collect coverage, clear coverage, run the
> python test, compare the two coverages, including branch coverage.

Attached is a small script for that. It runs two tests from zeroed
counters and lists, per function, the lines only one of them hits. It
needs a -Db_coverage=true build and lcov. It prints a Markdown table
by default, or plain text with --format text.

nocfbot-covdiff.py --format text --top 8 \
test_aio/001_aio test_aio/test_001_aio

perl = test_aio/001_aio
python = test_aio/test_001_aio

Lines only in one test, per function, largest first.
Showing 8 of 408 functions, use --top N for more, --top 0 for all.

perl python file:function
130 psql/mainloop.c:MainLoop
126 tcop/postgres.c:exec_bind_message
121 psql/startup.c:main
121 fe_utils/psqlscan.l:(no function)
84 psql/common.c:ExecQueryAndProcessResults
84 tcop/postgres.c:exec_simple_query
78 nodes/copyfuncs.c:copyObjectImpl
71 tcop/postgres.c:exec_execute_message

Lines hit: perl 68887, python 68420
Lines only in it: perl 2372, python 1893

On 001_aio from v1, the lines only the Perl test hits are psql, libpq
and the simple query path, and the rest of the suite covers all of
them. One real difference: Cluster.pm sets wal_level = minimal by
default and pypg does not, so the Python test misses the WAL skip path,
like smgrDoPendingSyncs().

Given we have not started migration yet, I am not sure whether a script like
this should go in the tree or live somewhere else. I would leave that to you
folks.

[1] https://postgr.es/m/aqxxGzx6Ll_sbIMH@paquier.xyz

Thanks,
Shihao