Re: Query execution in Perl TAP tests needs work

From: Noah Misch <noah(at)leadboat(dot)com>
To: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Cc: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Query execution in Perl TAP tests needs work
Date: 2023-08-29 01:48:19
Message-ID: 20230829014819.GB3575786@rfd.leadboat.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Aug 28, 2023 at 05:29:56PM +1200, Thomas Munro wrote:
> CI shows Windows
> consuming 4 CPUs at 100% for a full 10 minutes to run a test suite
> that finishes in 2-3 minutes everywhere else with the same number of
> CPUs. Could there be an event handling snafu in IPC::Run or elsewhere
> nearby? It seems like there must be either a busy loop or a busted
> sleep/wakeup... somewhere?

That smells like this one:
https://github.com/cpan-authors/IPC-Run/issues/166#issuecomment-1288190929

> As an experiment, I hacked up a not-good-enough-to-share experiment
> where $node->safe_psql() would automatically cache a BackgroundPsql
> object and reuse it, and the times for that test dropped ~51 -> ~9s on
> Windows, and ~7 -> ~2s on the Unixen.

Nice!

> suppose there are quite a few ways we could do better:
>
> 1. Don't fork anything at all: open (and cache) a connection directly
> from Perl.
> 1a. Write xsub or ffi bindings for libpq. Or vendor (parts) of the
> popular Perl xsub library?
> 1b. Write our own mini pure-perl pq client module. Or vendor (parts)
> of some existing one.
> 2. Use long-lived psql sessions.
> 2a. Something building on BackgroundPsql.
> 2b. Maybe give psql or a new libpq-wrapper a new low level stdio/pipe
> protocol that is more fun to talk to from Perl/machines?

(2a) seems adequate and easiest to achieve. If DBD::Pg were under a
compatible license, I'd say use it as the vendor for (1a). Maybe we can get
it relicensed? Building a separate way of connecting from Perl would be sad.

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Geoghegan 2023-08-29 03:18:22 Re: Logger process and "LOG: could not close client or listen socket: Bad file descriptor"
Previous Message Peter Geoghegan 2023-08-29 01:43:46 Re: Eager page freeze criteria clarification