PG_TEST_EXTRAs by theme rather than test name (Re: pgsql: Add tests for libpq gssencmode and sslmode options)

(moved to pgsql-hackers, change subject)

On 10/04/2024 18:54, Heikki Linnakangas wrote:
> On 10/04/2024 17:48, Peter Eisentraut wrote:
>> On 08.04.24 01:50, Heikki Linnakangas wrote:
>>> Add tests for libpq gssencmode and sslmode options
>>
>> Why aren't these tests at
>> src/interfaces/libpq/t/nnn_negotiate_encryption.pl ?
>
> To be honest, it never occurred to me. It started out as extra tests
> under src/test/ssl/, and when I decided to move them out to its own
> module, I didn't think of moving them to src/interfaces/libpq/t/.
>
> I will move it, barring any objections or better ideas.

Moved.

I also added an extra check for PG_TEST_EXTRA=kerberos, so that the
tests that require a MIT Kerberos installation are only run if
PG_TEST_EXTRA=kerberos is specified. That seems prudent; it seems
unlikely that you would want to run libpq_encryption tests with Kerberos
tests included, but not the main kerberos tests. If you specify
PG_TEST_EXTRA=libpq_encryption, but not 'kerberos', it's probably
because you don't have an MIT Kerberos installation on your system.

I added documentation for the new PG_TEST_EXTRA=libpq_encryption option,
I missed that earlier, with a note on the above interaction with 'kerberos'.

As we accumulate more PG_TEST_EXTRA options, I think we should
categorize the tests by the capabilities they need or the risk
associated, rather than by test names. Currently we have:

- kerberos: Requires MIT Kerberos installation and opens TCP/IP listen
sockets
- ldap: Requires OpenLDAP installation and opens TCP/IP listen sockets
- ssl: Opens TCP/IP listen sockets.
- load_balance: Requires editing the system 'hosts' file and opens
TCP/IP listen sockets.
- libpq_encryption: Opens TCP/IP listen sockets. For the GSSAPI tests,
requires MIT Kerberos installation
- wal_consistency_checking: is resource intensive
- xid_wraparound: is resource intensive

There are a few clear themes here:

- tests that open TCP/IP listen sockets
- tests that require OpenLDAP installation
- tests that require MIT Kerberos installation
- tests that require editing 'hosts' file
- tests that are resource intensive

We could have PG_TEST_EXTRA options that match those themes, and
enable/disable the individual tests based on those requirements. For
example, if you're on a single-user system and have no issue with
opening TCP/IP listen sockets, you would specify
"PG_TEST_EXTRA=tcp-listen", and all the tests that need to open TCP/IP
listen sockets would run. Also it would be nice to have autoconf/meson
tests for the presence of OpenLDAP / MIT Kerberos installations, instead
of having to enable/disable them with PG_TEST_EXTRA.

--
Heikki Linnakangas
Neon (https://neon.tech)