Re: More efficient build farm animal wakeup?

On Thu, Nov 24, 2022 at 10:00 AM Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> Are you saying you still think it's worth pursuing longpoll or similar methods for it, or that this is good enough?

I personally think it'd be pretty neat, to squeeze out that last bit
of latency. Maybe it's overkill...

The best idea I have so far for how to make it work more like a
service but still require nothing more than cron (so it's not hard for
people on systems where they don't even have root) is to have it start
if not already running (current lock file scheme already does that)
AND if some file buildfarm_enabled exists, or buildfarm_disabled
doesn't exist or something like that, and then keep running while
that's true. So if you need to turn it off for a while you can just
touch/rm that, but normally it'll keep running its wait loop forever,
and start up soon after a reboot; maybe it also exits if you touch the
config file so it can restart next time and reread it, or something
like that. Then it can spend all day in a loop that does 120s long
polls, and start builds within seconds of a new commit landing.

Curious to know how you'd build the server side. You mentioned a
commit hook notifying some kind of long poll distributor. Would you
use a Twisted/async/whatever-based server that knows how to handle
lots of sockets efficiently, or just use old school web server tech
that would block waiting for NOTIFY or something like that? You'd
probably get away with that for the small numbers of animals involved
(I mean, a couple of hundred web server threads/processes just sitting
there waiting would be borderline acceptable I guess). But it'd be
more fun to do it with async magic.