Re: Probable CF bot degradation

On Wed, Mar 23, 2022 at 12:44:09PM +1300, Thomas Munro wrote:
> On Mon, Mar 21, 2022 at 12:46 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> > On Mon, Mar 21, 2022 at 12:23 PM Thomas Munro <thomas(dot)munro(at)gmail(dot)com> wrote:
> > > On Mon, Mar 21, 2022 at 1:58 AM Matthias van de Meent <boekewurm+postgres(at)gmail(dot)com> wrote:
> > > > Additionally, are there plans to validate commits of the main branch
> > > > before using them as a base for CF entries, so that "bad" commits on
> > > > master won't impact CFbot results as easy?
> > >
> > > How do you see this working?
> >
> > [Now with more coffee on board] Oh, right, I see, you're probably
> > thinking that we could look at
> > https://github.com/postgres/postgres/commits/master and take the most
> > recent passing commit as a base. Hmm, interesting idea.
>
> A nice case in point today: everything is breaking on Windows due to a
> commit in master, which could easily be avoided by looking back a
> certain distance for a passing commit from postgres/postgres to use as
> a base. Let's me see if this is easy to fix...
>
> https://www.postgresql.org/message-id/20220322231311.GK28503%40telsasoft.com

I suggest not to make it too sophisticated. If something is broken, the CI
should show that rather than presenting a misleading conclusion.

Maybe you could keep track of how many consecutive, *new* failures there've
been (which were passing on the previous run for that task, for that patch) and
delay if it's more than (say) 5. For bonus points, queue a rerun of all the
failed tasks once something passes.

If you create a page to show the queue, maybe it should show the history of
results, too. And maybe there should be a history of results for each patch.

If you implement interactive buttons, maybe it could allow re-queueing some
recent failures (add to end of queue).

--
Justin