Re: [PoC PATCH] Parallel dump to /dev/null

From: Michael Banck <michael(dot)banck(at)credativ(dot)de>
To: pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: [PoC PATCH] Parallel dump to /dev/null
Date: 2018-02-28 13:28:41
Message-ID: 20180228132840.GD13784@nighthawk.caipicrew.dd-dns.de
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hi,

On Thu, Feb 01, 2018 at 02:24:46PM +0100, Michael Banck wrote:
> dumping a database to /dev/null via pg_dump is (AFAIK) one recommended
> way to check for corruption. However, dumping to /dev/null is currently
> not supported in directory mode which makes it not possible to dump to
> /dev/null in parallel.
>
> I had a look at this, and it appears it would suffice to just override
> the few spots in pg_backup_directory.c which assemble filenames in the
> target directory to revert to '/dev/null' (or 'NUL' on Windows).
>
> The attached proof-of-concept patch does that, and it seems to work; I'm
> getting some nice speedups on a 170 GB test database:
>
> $ time pg_dump -Fc -Z0 -f /dev/null TESTDB
> real 32m45.120s
> [...]
> $ time pg_dump -Fd -j8 -Z0 -f /dev/null TESTDB
> real 9m28.357s

I have added this patch to the next commitfest:

https://commitfest.postgresql.org/17/1576/

I also tried to add a TAP test, but as this patch produces no dump
output, I had to exclude it from all restores tests and then got an
off-by-one between the number of tests that were expected vs. those that
were run. I've attached it if somebody wants to take a look, but I hope
with Stephen's refactoring of the pg_dump TAP tests, this might be
easier.

Michael

Attachment Content-Type Size
0001-WIP-pg_dump-TAP-testcase.patch text/x-diff 23.6 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Amit Kapila 2018-02-28 13:32:47 Re: Contention preventing locking
Previous Message Alexander Kuzmenkov 2018-02-28 12:55:19 Re: [patch] BUG #15005: ANALYZE can make pg_class.reltuples inaccurate.