Re: Heads Up: cirrus-ci is shutting down June 1st

Hi,

Attached is an large incremental patch onto Bilal's version:

- I hacked enough on the pg-vm-images repo to make it store containers in
github (on the gha_main branch, for now). That makes the container faster
and cheaper to retrieve.

We might want to split the containers further (e.g. cross building and 32bit
support), but for now I just split the docs stuff into a separate container.

- The reason that sometimes cancelling took a long time was, afaict, the use
of always() in the compiler-warning job. That apparently prevents GHA from
cancelling the job in a timely fashion. I turned those into !cancelled().

Also addded a !cancelled() to SanityCheck.

- In the end I didn't like the matrix all that much, makes it considerably
harder to understand everything and the restrictions GHA puts on it are just
too annoying.

I replaced it much more heavy use of yaml anchors/aliases and by updating
the environment programattically. I'm not sure how much better it is now.

- The macports cache keys included the run_id, I think that's a bad idea,
because it leads to the cache being newly uploaded even if there's been no
change. That'll lead to even more quickly churning through the cache space.

- ccache:

- There was too much duplication around the ccache handling for my taste. I
moved that into a yaml anchor and reused it everywhere. By using
${{github.job_id}} the cache names don't need to be manually disambiguated.

- The ccache names weren't unique enough. The run_id doesn't change during
reruns which would lead to warnings.

- Made it so that the cache is saved immediately after the build, so that
cancelled builds still save the cache.

- I wanted to share commands like meson test between the tasks, made that work
with a bit of hackery.

- I didn't see why
find / -maxdepth 1 -type f -name 'core*' -exec mv '{}' /tmp/cores/ \;
was needed.

- I found it hard to find actual warnings in the output of CompilerWarnings,
due to to all the configure output. I added some magic output stuff to make
the configure output collapse once it ran.

- I converted most things to use ${{env.varname}} instead of $VARNAME or
%VARNAME%, as that allows sharing code between windows and other OSs if one
is a bit careful.

- Deduplicated a few other things like logging.

- Added a commit removing cirrus, mainly because I was tired of it also
running while hacking on this.

- Also added a commit to reduce the segment size in tests to 1MB, that makes
them a decent bit less IO intensive.

- Added a commit to just run regress/regress, makes it a lot faster to test
"complete-ish" cycles.

- Changed sanitizer using builds to use -O2, to reduce the CPU cost a bit.

This makes uncached builds noticeably slower, but does appear to be a
win. But I could see counter-arguments to that too.

- Removed :detect_stack_use_after_return=0, as that's been made unnecessary
since you "forked off" from .cirrus.tasks.yml.

A few other comments:

- All the caches in GHA apparently are branch specific, except that branches
can access the caches of the main branch.

I think that'll make particularly macports very expensive for cfbot. I
wonder if we ought to build the "base" macports cache in pg-vm-images.

Similarly, I think we probably should do that for mingw64.

- src/tools/ci/README needs updating

- I experimented with making the slowest jobs faster by using
meson test --slice 1/2
that actually does nicely work. But it didn't quite seem critical for now.

Here's a run running with just regress/regress:
https://github.com/anarazel/postgres/actions/runs/26782558353
(the CompilerWarnings ccache hit had a miss, that's why it's not faster)

Here's a (not yet completed) run with the full tests:
https://github.com/anarazel/postgres/actions/runs/26784248887

Thoughts?

Greetings,

Andres Freund