Re: CI and test improvements

On Sun, Oct 02, 2022 at 01:52:01PM -0700, Andres Freund wrote:
> Hi,
>
> On 2022-10-01 18:36:41 -0700, Andres Freund wrote:
> > I am wondering if we should instead introduce a new "quickcheck" task that
> > just compiles and runs maybe one test and have *all* other tests depend on
> > that. Wasting a precious available windows instance to just fail to build or
> > immediately fail during tests doesn't really make sense.

> With a primed cache this takes ~32s, not too bad imo. 12s of that is
> cloning the repo.

Maybe - that would avoid waiting 4 minutes for a windows instance to
start in the (hopefully atypical) case of a patch that fails in 1-2
minutes under linux/freebsd.

If the patch were completely broken, the windows task would take ~4min
to start, plus up to ~4min before failing to compile or failing an early
test. 6-8 minutes isn't nothing, but doesn't seem worth the added
complexity.

Also, this would mean that in the common case, the slowest task would be
delayed until after the SanityCheck task instance starts, compiles, and
runs some test :( Your best case is 32sec, but I doubt that's going to
be typical.

I was thinking about the idea of cfbot handling "tasks" separately,
similar to what it used to do with travis/appveyor. The logic for
"windows tasks are only run if linux passes tests" could live there.
That could also be useful if there's ever the possibility of running an
additional OS on another CI provider, or if another provider can run
windows tasks faster, or if we need to reduce our load/dependency on
cirrus. I realized that goes backwards in some ways to the direction
we've gone with cirrus, and I'm not sure how exactly it would do that (I
suppose it might add ci-os-only tags to its commit message).

> + # no options enabled, should be small
> + CCACHE_MAXSIZE: "150M"

Actually, tasks can share caches if the "cache key" is set.

If there was a separate "Sanity" task, I think it should use whatever
flags linux (or freebsd) use to avoid doing two compilations (with lots
of cache misses for patches which modify *.h files, which would then
happen twice, in serial).

> + # use always: to continue after failures. Task that did not run count as a
> + # success, so we need to recheck SanityChecks's condition here ...

> - # task that did not run, count as a success, so we need to recheck Linux'
> - # condition here ...

Another/better justification/description is that "cirrus warns if the
depending task has different only_if conditions than the dependant task".

--
Justin