Re: More efficient build farm animal wakeup?

On 2022-11-20 Su 17:32, Thomas Munro wrote:
> On Sun, Nov 20, 2022 at 2:44 AM Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> It might not suit your use case, but one of the things I do to reduce
>> fetch load is to run a local mirror which runs
>>
>> git fetch -q --prune
>>
>> every 5 minutes. It also runs a git daemon, and several of my animals
>> point at that.
> Thanks. I understand now that my configuration without a local mirror
> is super inefficient (it spends the first ~25s of each minute running
> git commands). Still, even though that can be improved by me setting
> up more stuff, I'd like something event-driven rather than short
> polling-based for lower latency.
>
>> If there's a better git API I'll be happy to try to use it.
> Cool. Seems like we just have to invent something first...
>
> FWIW I'm also trying to chase the short polling out of cfbot. It
> regularly harasses the git servers at one end (could be fixed with
> this approach), and wastes a percentage of our allotted CPU slots on
> the other end by scheduling periodically (could be fixed with webhooks
> from Cirrus).

I think I have solved most of the actual issues without getting too complex.

Here's how:

The buildfarm server now creates a companion to branches_of_interest.txt
called branches_of_interest.json which looks like this:

[
   {
      "REL_11_STABLE" : "140c803723"
   },
   {
      "REL_12_STABLE" : "4cbcb7ed85"
   },
   {
      "REL_13_STABLE" : "c13667b518"
   },
   {
      "REL_14_STABLE" : "5cda142bb9"
   },
   {
      "REL_15_STABLE" : "ff9d27ee2b"
   },
   {
      "HEAD" : "51b5834cd5"
   }
]

It updates this every time it does a git fetch, currently every 5 minutes.

run_branches.pl fetches this file instead of the plain list of branches,
and before running run_build.pl checks if the given commit was the
latest one tested, and if so and a build isn't being forced, skips the
branch. Thus, in the case where all the branches are up to date there
will be no git calls whatsoever.

You can try it out by getting run_branches.pl from
<https://raw.githubusercontent.com/PGBuildFarm/client-code/main/run_branches.pl>

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com