reduce size of logs stored by buildfarm

From: Álvaro Herrera <alvherre(at)kurilemu(dot)de>
To: Pg Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: reduce size of logs stored by buildfarm
Date: 2025-11-25 13:30:33
Message-ID: 202511251218.zfs4nu2qnh2m@alvherre.pgsql
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

I just noticed that one regress log file for the pg_dump test is about
7MB long because it contains totally unnecessary dump files in the
regress log output. This is because the tests run pg_dump of schema
pg_catalog without redirecting the output anywhere, which means it goes
to stdout, and then `prove` helpfully puts it in the regress log file.
This is completely pointless. We can make the log file 1 MB instead by
adding a --file parameter to that command, as 0001 attached.

We also have a couple of tests that leave rather large server logs
behind that are perhaps not very useful. I would propose to run those
servers with log_statement=off and avoid some of the bloat. 0002 does
that.

Another possible approach, instead of individually tweaking t/*.pl files
to add 'log_statements=none', would be to set it that way in
PostgreSQL::Test::Cluster globally. Right now we have
log_statements=all there, but maybe that was not a great idea.

I also noticed that an oauth test file contains a couple MBs of a
gigantic string of just 'x'. I suppose that will compress well (with
gzip at least, the 2 MB file becomes 8 kB). Still, it's kinda
ridiculous and useless to bloat a 67 kB file to 2 MB that way. The
problem here is that we print 'sending JSON response' and the verbatim
response, included the 1 MB of "x" as '_pad_'. Can we supress this?
0003 does that by simply cutting the string short at 10k, which reduces
the size of the log from 2 MB to some 86 kB. Maybe there are better
ways to deal with this? Jacob?

With these three patches, we go from having 62 MB bytes in files with
"log" in their names under testrun/ (except those that have "catalog")
to 30 MB. I think that's not a bad reduction.

--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
Are you not unsure you want to delete Firefox?
[Not unsure] [Not not unsure] [Cancel]
http://smylers.hates-software.com/2008/01/03/566e45b2.html

Attachment Content-Type Size
0001-pg_dump-tests-don-t-put-dumps-in-stdout.patch text/x-diff 1.2 KB
0002-log_statement-off.patch text/x-diff 3.1 KB
0003-oauth_server-don-t-log-huge-strings.patch text/x-diff 1.1 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message 河田達也 2025-11-25 13:48:06 Re: [Proposal] Adding TRIM_SPACE option to COPY
Previous Message Viktor Holmberg 2025-11-25 13:24:17 Re: ON CONFLICT DO SELECT (take 3)