Re: More efficient build farm animal wakeup?

On 2022-11-22 Tu 13:04, Magnus Hagander wrote:
>
>
> On Tue, Nov 22, 2022 at 12:10 AM Magnus Hagander <magnus(at)hagander(dot)net>
> wrote:
>
>
>
> On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan
> <andrew(at)dunslane(dot)net> wrote:
>
>
> On 2022-11-21 Mo 16:20, Magnus Hagander wrote:
> > n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
> wrote:
> >
> >     Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> b>     > The buildfarm server now creates a companion to
> >     branches_of_interest.txt
> >     > called branches_of_interest.json which looks like this:
> >
> >     ... okay ...
> >
> >
> > Yeah, it's not as efficient as something like long polling
> or web
> > sockets, but it is most definitely a lot simpler!
> >
> > If we're going to have a lot of animals do pulls of this
> file every
> > minute or more, it's certainly a lot better to pull this
> small file
> > than to make multiple git calls.
> >
> > It could trivially be made even more efficient by making the
> request
> > with either a If-None-Match or If-Modified-Since. While it's
> still
> > small, that cuts the size approximately in half, and would
> allow you
> > to skip even more processing if nothing has changed.
>
>
> I'll look at that.
>
>
> >
> >
> >     > It updates this every time it does a git fetch,
> currently every
> >     5 minutes.
> >
> >     That up-to-five-minute delay, on top of whatever cronjob
> delay one has
> >     on one's animals, seems kind of sad.  I've gotten kind
> of spoiled
> >     maybe
> >     by seeing first buildfarm results typically within 15
> minutes of a
> >     push.
> >     But if we're trying to improve matters in this area,
> this doesn't seem
> >     like quite the way to go.
> >
> >     But it does seem like this eliminates one expense.  Now
> that you have
> >     that bit, maybe we could arrange a webhook or something
> that allows
> >     branches_of_interest.json to get updated immediately
> after a push?
> >
> >
> > Webhooks are definitely a lot easier to implement in between our
> > servers yeah, so that shouldn't be too hard. We could use
> the same
> > hooks that we use for borka to build the docs, but have it
> just run
> > whatever script it is the buildfarm needs. I assume it's just
> > something trivial to run there, Andrew?
>
>
> Yes, I think much better between servers. Currently the cron
> job looks
> something like this:
>
>
> */5 * * * * cd $HOME/postgresql.git && git fetch -q &&
> $HOME/website/bin/branches_of_interest.pl
> <http://branches_of_interest.pl>
>
>
> That script is what sets up the json files.
>
>
> I know nothing about git webhooks though, someone will have to
> point me
> in the right direction.
>
>
> I can set that up for you -- we have ready-made packages for 95%
> of what's needed for that one as we use it elsewhere in the infra.
> So I'll just set something up that will run that exact script (as
> the correct user of course) and comment out the cronjob,and then
> send you the details of what is set up where (I don't recall it
> offhand, but as it's the same we have elsewhere I'll find it
> quickly once I look into it).
>  
>
>
> Hi!
>
> This should now be set up, and Andrew has been sent the instructions
> for how to access that setup on the buildfarm server. So hopefully it
> will now be updating the buildfarm server side of things within a
> couple of seconds from a commit, and not do any speculative pulls. But
> we'll keep an extra eye on it for a bit of course, as it's entirely
> possible I got something worng :)
>
> (This is only the part git -> bf server, of course, as that step
> doesn't need any client changes it was easier to do quickly)
>
>

The server side appears to be working well.

The new client side code is being tested on crake and working fine - the
all-up-to-date case takes just a second or two, almost all of which is
taken with getting the json file from the server. No git calls at all
are done on the client in this case.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com