Re: [Pgbuildfarm-members] [HACKERS] Reducing buildfarm disk usage: remove temp installs when done

On 01/21/2015 12:43 AM, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> On 01/20/2015 10:44 PM, Tom Lane wrote:
>>> Erm ... no patch attached?
>> Oh, darn. Here it is.
> I've applied this on dromedary, and it definitely makes a nice dent
> in the at-rest space consumption for a full set of branches. The old
> data directory contents were
>
> $ du -hsc pgmirror.git HEAD REL*
> 418M pgmirror.git
> 367M HEAD
> 348M REL9_0_STABLE
> 353M REL9_1_STABLE
> 356M REL9_2_STABLE
> 360M REL9_3_STABLE
> 277M REL9_4_STABLE
> 2.4G total
>
> Post-patch, with a freshly created data directory (including a
> fresh clone from the git server), I've got
>
> $ du -hsc pgmirror.git HEAD REL*
> 167M pgmirror.git
> 107M HEAD
> 86M REL9_0_STABLE
> 91M REL9_1_STABLE
> 95M REL9_2_STABLE
> 100M REL9_3_STABLE
> 105M REL9_4_STABLE
> 753M total
>
> It appears that the peak transient space consumption while building a
> branch is about 500MB. In addition to these numbers, I've got a shade
> under 1GB in ccache space (that's configurable of course, but by default
> ccache will eat up to that much). So the total disk space to run a
> buildfarm member with ccache was something close to 4GB with the old way,
> 2.25GB with this patch.
>
> (These numbers aren't totally comparable of course, since the year-old
> installation had acquired some git repo bloat which this one hasn't
> had time to yet. But any way you slice it, I've saved well more than
> 1GB of space.)
>
>

Some more thoughts about this:

* there is probably precious little virtue in keeping a local git
mirror for most use cases, once we do this
* regular running of "git gc" pays handsomely, by the look of it. Not
sure if we should have the buildfarm client do this or just make it
a cron job.
* I'm experimenting with removing the work tree on success and
checking it out again when we run. That reduces the static storage
per branch to a handful of megabytes, at the cost of a little IO and
a few seconds of processing time.
* vpath builds also save space and time, since we don't copy the
source in that case.

cheers

andrew