Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

Hi,

On 2023-08-08 16:28:49 +0200, Peter Eisentraut wrote:
> On 08.08.23 04:15, Andres Freund wrote:
> > Potential paths forward for cfbot, in addition to the above:
> >
> > - Pay for compute / ask the various cloud providers to grant us compute
> > credits. At least some of the cloud providers can be used via cirrus-ci.
> >
> > - Host (some) CI runners ourselves. Particularly with macos and windows, that
> > could provide significant savings.
> >
> > - Build our own system, using buildbot, jenkins or whatnot.
>
> I think we should use the "compute credits" plan from Cirrus CI. It should
> be possible to estimate the costs for that. Money is available, I think.

Unfortunately just doing that seems like it would up considerably on the too
expensive side. Here are the stats for last months' cfbot runtimes (provided
by Thomas):

task_name | sum
------------------------------------------------+------------
FreeBSD - 13 - Meson | 1017:56:09
Windows - Server 2019, MinGW64 - Meson | 00:00:00
SanityCheck | 76:48:41
macOS - Ventura - Meson | 873:12:43
Windows - Server 2019, VS 2019 - Meson & ninja | 1251:08:06
Linux - Debian Bullseye - Autoconf | 830:17:26
Linux - Debian Bullseye - Meson | 860:37:21
CompilerWarnings | 935:30:35
(8 rows)

If I did the math right, that's about 7000 credits (and 1 credit costs 1 USD).

task costs in credits
linux-sanity: 55.30
linux-autoconf: 598.04
linux-meson: 619.40
linux-compiler-warnings: 674.28
freebsd : 732.24
windows : 1201.09
macos : 3143.52

Now, those times are before optimizing test runtime. And besides optimizing
the tasks, we can also optimize not running tests for docs patches etc. And
optimize cfbot to schedule a bit better.

But still, the costs look not realistic to me.

If instead we were to use our own GCP account, it's a lot less. t2d-standard-4
instances, which are faster than what we use right now, cost $0.168984 / hour
as "normal" instances and $0.026764 as "spot" instances right now [1]. Windows
VMs are considerably more expensive due to licensing - 0.184$/h in addition.

Assuming spot instances, linux+freebsd tasks would cost ~100USD month (maybe
10-20% more in reality, due to a) spot instances getting terminated requiring
retries and b) disks).

Windows would be ~255 USD / month (same retries caveats).

Given the cost of macos, it seems like it'd be by far the most of affordable
to just buy 1-2 mac minis (2x ~660USD) and stick them in a shelf somewhere, as
persistent runners. Cirrus has builtin macos virtualization support - but can
only host two VMs on each mac, due to macos licensing restrictions. A single
mac mini would suffice to keep up with our unoptimized monthly runtime
(although there likely would be some overhead).

Greetings,

Andres Freund

[1] https://cloud.google.com/compute/all-pricing