Re: More efficient build farm animal wakeup?

On Tue, Nov 22, 2022 at 12:10 AM Magnus Hagander <magnus(at)hagander(dot)net>
wrote:

>
>
> On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan <andrew(at)dunslane(dot)net>
> wrote:
>
>>
>> On 2022-11-21 Mo 16:20, Magnus Hagander wrote:
>> > n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> >
>> > Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> b> > The buildfarm server now creates a companion to
>> > branches_of_interest.txt
>> > > called branches_of_interest.json which looks like this:
>> >
>> > ... okay ...
>> >
>> >
>> > Yeah, it's not as efficient as something like long polling or web
>> > sockets, but it is most definitely a lot simpler!
>> >
>> > If we're going to have a lot of animals do pulls of this file every
>> > minute or more, it's certainly a lot better to pull this small file
>> > than to make multiple git calls.
>> >
>> > It could trivially be made even more efficient by making the request
>> > with either a If-None-Match or If-Modified-Since. While it's still
>> > small, that cuts the size approximately in half, and would allow you
>> > to skip even more processing if nothing has changed.
>>
>>
>> I'll look at that.
>>
>>
>> >
>> >
>> > > It updates this every time it does a git fetch, currently every
>> > 5 minutes.
>> >
>> > That up-to-five-minute delay, on top of whatever cronjob delay one
>> has
>> > on one's animals, seems kind of sad. I've gotten kind of spoiled
>> > maybe
>> > by seeing first buildfarm results typically within 15 minutes of a
>> > push.
>> > But if we're trying to improve matters in this area, this doesn't
>> seem
>> > like quite the way to go.
>> >
>> > But it does seem like this eliminates one expense. Now that you
>> have
>> > that bit, maybe we could arrange a webhook or something that allows
>> > branches_of_interest.json to get updated immediately after a push?
>> >
>> >
>> > Webhooks are definitely a lot easier to implement in between our
>> > servers yeah, so that shouldn't be too hard. We could use the same
>> > hooks that we use for borka to build the docs, but have it just run
>> > whatever script it is the buildfarm needs. I assume it's just
>> > something trivial to run there, Andrew?
>>
>>
>> Yes, I think much better between servers. Currently the cron job looks
>> something like this:
>>
>>
>> */5 * * * * cd $HOME/postgresql.git && git fetch -q &&
>> $HOME/website/bin/branches_of_interest.pl
>>
>>
>> That script is what sets up the json files.
>>
>>
>> I know nothing about git webhooks though, someone will have to point me
>> in the right direction.
>>
>
> I can set that up for you -- we have ready-made packages for 95% of what's
> needed for that one as we use it elsewhere in the infra. So I'll just set
> something up that will run that exact script (as the correct user of
> course) and comment out the cronjob,and then send you the details of what
> is set up where (I don't recall it offhand, but as it's the same we have
> elsewhere I'll find it quickly once I look into it).
>
>

Hi!

This should now be set up, and Andrew has been sent the instructions for
how to access that setup on the buildfarm server. So hopefully it will now
be updating the buildfarm server side of things within a couple of seconds
from a commit, and not do any speculative pulls. But we'll keep an extra
eye on it for a bit of course, as it's entirely possible I got something
worng :)

(This is only the part git -> bf server, of course, as that step doesn't
need any client changes it was easier to do quickly)

//Magnus