Re: ci: CCache churns through available space too quickly

From: "Tristan Partin" <tristan(at)partin(dot)io>
To: "Andres Freund" <andres(at)anarazel(dot)de>, "Nazir Bilal Yavuz" <byavuz81(at)gmail(dot)com>, "Jacob Champion" <jacob(dot)champion(at)enterprisedb(dot)com>, "Thomas Munro" <thomas(dot)munro(at)gmail(dot)com>
Cc: "pgsql-hackers" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: ci: CCache churns through available space too quickly
Date: 2026-06-17 22:02:09
Message-ID: DJBNW7DKLMV1.2WHERLY9NSEJX@partin.io
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri Jun 5, 2026 at 8:09 PM UTC, Andres Freund wrote:
> Hi,
>
> I noticed that a handfull of CI runs already lead to exceeding the available
> cache space. One can pay for more cache space, but I think the problem is
> more that what we currently do doesn't work well.
>
> With cirrus-ci all branches shared one cache, but that's not the case with
> github actions. Except for being able to read caches from the default branch
> (master in our case), other branches have completely separate cache
> namespaces. That's probably the right call, safety wise, but makes our ccache
> approach .. not great.
>
> We should only upload a new cache when the ccache cache hit ratio of the
> existing cache entry has gotten low.

I had started reviewing this patch the day it was originally sent, but
due to circumstances I couldn't finish the review before it was
committed. I had some thoughts with regard to improving the Python
script itself. Attached are some improvements that make the code
a little more pythonic as well as more easily usable locally for testing
purposes. Some of the patches may be more valuable than others.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)

Attachment Content-Type Size
v1-0001-Use-long-options-for-ccache-commands.patch text/x-patch 1.6 KB
v1-0002-Use-json-to-parse-ccache-statistics.patch text/x-patch 1.9 KB
v1-0003-Remove-the-shutil.which-call.patch text/x-patch 1.4 KB
v1-0004-Use-a-context-manager-to-write-grouped-CI-output.patch text/x-patch 2.9 KB
v1-0005-Provide-ccache-target-rate-as-an-input-to-the-scr.patch text/x-patch 4.1 KB
v1-0006-Check-that-GITHUB_OUTPUT-exists-before-trying-to-.patch text/x-patch 1.4 KB
v1-0007-Make-gha_ccache_decide.py-executable.patch text/x-patch 675 bytes

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Jacob Champion 2026-06-17 22:04:34 Re: Require SSL connection to postgres for oauth
Previous Message Noah Misch 2026-06-17 21:59:52 Re: Non-text mode for pg_dumpall