| From: | Pavlo Sukhanov <pavlo(at)toothywalrus(dot)dev> |
|---|---|
| To: | me(at)jeltef(dot)nl |
| Cc: | nik(at)postgres(dot)ai, pgsql-hackers(at)lists(dot)postgresql(dot)org |
| Subject: | Re: Python/pytest test framework take two |
| Date: | 2026-11-22 10:06:09 |
| Message-ID: | CA+c-BchZOt0xN_hsV2-XBujCpmT4ymQ+zU=AhPxyeXTLDE66iw@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi Jelte,
This is my first review on this list. I went through v3 (all 12
patches), applied on master 77bba54 (2026-11-13), and tested it on
Ubuntu 24.04 in containers. Overall it works well, and the API was
easy to pick up as a newcomer: the pg.sql() result shapes,
LibpqError.sqlstate, and NOTICEs as warnings that pytest.warns can
catch. Below is what I tested and what I ran into, roughly in order of
how much I think it matters.
What I tested
- Everything applies except the .github/workflows/pg-ci.yml hunk in
0004 (same as cfbot). Since 24723e7c (2026-11-18), which added section
11e to 049_wait_for_lsn.pl, 0008 doesn't apply either, and 0010/0011
then fail on recovery/meson.build. The Python port of 049 will need
11e too. Also, since 6b0896f3 (same day) CI passes server settings
through the new PG_TEST_SERVER_POSTGRES_OPTS, which v3 doesn't read
yet, so after a rebase the io_uring jobs wouldn't apply
io_method=io_uring to the pytest tests.
- meson, Python 3.12, pytest 7.4.4 on aarch64: all 7 ported tests
pass, and the full meson suite is 389 OK, 0 failures. The 7 ports also
pass on x86_64.
- autoconf: ./configure --enable-pytest, then make check: all 7 pass.
- Timing against the Perl originals (median of 3 runs, aarch64): about
13% faster in total. Clear wins for amcheck/001 (-45%) and
subscription/031_column_list (-58%); no measurable difference for 049,
recovery/031 and 002_inline. That's the same pattern as the CI numbers
in the commit messages. I can send the table if it's useful.
Issues
1. A meson timeout or SIGTERM leaves servers running. "meson test -t
0.005 recovery/test_049_wait_for_lsn" times out and leaves the primary
and standby postmasters (with walsender and walreceiver) running. The
Perl version of the same test leaves nothing behind, thanks to the die
handler at Cluster.pm:181 and the END block. Ctrl-C does clean up. On
CI and buildfarm timeouts that means leftover ports, shared memory and
data directories. A SIGTERM handler that raises, so fixture teardown
runs, or a finalizer that does "pg_ctl stop -m immediate" would
probably cover it.
2. Authentication differs between template-copied and freshly initdb'd
servers. The template is created with trust, but when initdb_opts is
given or PG_TEST_INITDB_EXTRA_OPTS is set, server.py (around line 225)
runs initdb with --auth peer. Then the README's own example,
pg.sql_oneshot("SELECT current_user", user="alice"), fails with "Peer
authentication failed", and a TCP-only node can't make its default
connection at all ("Ident authentication failed").
PG_TEST_INITDB_EXTRA_OPTS is the variable regress.sgml suggests for
running the tests with extra initdb options, so a test like that would
pass in a plain run and fail for anyone who uses it. The Perl
framework always uses trust. Would it make sense to do the same here,
or to reset pg_hba the same way on both paths?
3. The declared minimum versions don't hold.
- @pytest.hookimpl(wrapper=True) needs pluggy >= 1.2. Bookworm ships
pytest 7.2.1 with pluggy 1.0.0, and with those versions every test
file fails at startup with "TypeError: __call__() got an unexpected
keyword argument 'wrapper'". (I pinned the versions with pip rather
than using a real Bookworm userland.)
- On Python 3.9, datetime.fromisoformat() rejects the "+00" offset
PostgreSQL sends, so test_001_aio and test_029_stats_restart fail.
According to the Python docs 3.10 has the same limitation; I didn't
run it. This may be the "known Python 3.9 timestamp failures" Nikolay
mentioned.
So either the floors go up (pluggy >= 1.2, Python >= 3.11), or the
code adapts (hookwrapper=True, normalizing the offset before parsing).
4. Three small autoconf issues:
- pytest_check starts with rm -rf tmp_check (Makefile.global.in),
which deletes the logs prove_check just wrote in the same directory.
- pytest_installcheck's PATH lacks $(CURDIR), unlike
prove_installcheck, so under make installcheck test_json_parser fails
156/156 with FileNotFoundError while the TAP test passes.
- configure's "$PYTHON" -m pytest fallback only works when PYTHON is
set, i.e. with --with-python or an explicit PYTHON=.
5. Teardown leaks a server when close() raises. PGconn.close() raises
UnconsumedResultError before PQfinish, and in the create_pg teardown
that skips stop() for that server and every later one; the postmaster
is still running after pytest exits. Putting stop() in a try/finally
would fix it.
Smaller things
- PG_CONFIG is run through /bin/sh (a single string passed to
util.run() means shell=True), so a path with a space fails with exit
status 127. It's the same 127 you get when tmp_install is missing,
which makes that error confusing.
- If a test catches the TimeoutError from .result(timeout=...) and the
query later fails, the error is never reported, because result() marks
the future consumed before waiting. That's the part of Nikolay's
report v3 doesn't cover. Low priority.
- README: "meson test --suite pytest" selects nothing (the tests are
in their directories' suites), and running pytest directly from the
source root needs pytest >= 8.4 or PYTHONPATH=src/test/pytest. The pg
fixture's docstring still says roles and databases are rolled back,
while the README says they persist.
- "pip install ." packages 26 stray modules as top-level test/ and
tools/ packages, since there's no [build-system] and setuptools
auto-discovers them.
I have repro scripts and logs for all of this and can send any of
them. I'm happy to test the next version on the same setups.
Thanks,
Pavlo Sukhanov
| From | Date | Subject | |
|---|---|---|---|
| Previous Message | Stepan Neretin | 2026-11-22 09:16:31 | Re: FATAL: could not send end-of-streaming message to primary: no COPY in progress |