| From: | "Bossart, Nathan" <bossartn(at)amazon(dot)com> |
|---|---|
| To: | Dipesh Pandit <dipesh(dot)pandit(at)gmail(dot)com> |
| Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>, Jeevan Ladhe <jeevan(dot)ladhe(at)enterprisedb(dot)com>, Stephen Frost <sfrost(at)snowman(dot)net>, Andres Freund <andres(at)anarazel(dot)de>, Hannu Krosing <hannuk(at)google(dot)com>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org> |
| Subject: | Re: .ready and .done files considered harmful |
| Date: | 2021-08-23 02:31:45 |
| Message-ID: | 3ECC212F-88FD-4FB2-BAF1-C2DD1563E310@amazon.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On 8/21/21, 9:29 PM, "Bossart, Nathan" <bossartn(at)amazon(dot)com> wrote:
> I was curious about this, so I wrote a patch (attached) to store
> multiple files per directory scan and tested it against the latest
> patch in this thread (v9) [0]. Specifically, I set archive_command to
> 'false', created ~20K WAL segments, then restarted the server with
> archive_command set to 'true'. Both the v9 patch and the attached
> patch completed archiving all segments in just under a minute. (I
> tested the attached patch with NUM_FILES_PER_DIRECTORY_SCAN set to 64,
> 128, and 256 and didn't observe any significant difference.) The
> existing logic took over 4 minutes to complete.
>
> I'm hoping to do this test again with many more (100K+) status files,
> as I believe that the v9 patch will be faster at that scale, but I'm
> not sure how much faster it will be.
I ran this again on a bigger machine with 200K WAL files pending
archive. The v9 patch took ~5.5 minutes, the patch I sent took ~8
minutes, and the existing logic took just under 3 hours.
Nathan
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Tom Lane | 2021-08-23 02:47:25 | Re: Improved regular expression error message for backrefs |
| Previous Message | Masahiko Sawada | 2021-08-23 01:46:42 | Re: Showing I/O timings spent reading/writing temp buffers in EXPLAIN |