[PoC PATCH] Parallel dump to /dev/null

From: Michael Banck <michael(dot)banck(at)credativ(dot)de>
To: pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: [PoC PATCH] Parallel dump to /dev/null
Date: 2018-02-01 13:24:46
Message-ID: 20180201132446.GA13784@nighthawk.caipicrew.dd-dns.de
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hi,

dumping a database to /dev/null via pg_dump is (AFAIK) one recommended
way to check for corruption. However, dumping to /dev/null is currently
not supported in directory mode which makes it not possible to dump to
/dev/null in parallel.

I had a look at this, and it appears it would suffice to just override
the few spots in pg_backup_directory.c which assemble filenames in the
target directory to revert to '/dev/null' (or 'NUL' on Windows).

The attached proof-of-concept patch does that, and it seems to work; I'm
getting some nice speedups on a 170 GB test database:

$ time pg_dump -Fc -Z0 -f /dev/null TESTDB
real 32m45.120s
[...]
$ time pg_dump -Fd -j8 -Z0 -f /dev/null TESTDB
real 9m28.357s

Thoughts?

Michael

--
Michael Banck
Projektleiter / Senior Berater
Tel.: +49 2166 9901-171
Fax: +49 2166 9901-100
Email: michael(dot)banck(at)credativ(dot)de

credativ GmbH, HRB Mönchengladbach 12080
USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer

Attachment Content-Type Size
parallel-pgdump-dev-null.patch text/x-diff 3.3 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Bruce Momjian 2018-02-01 13:32:59 Re: proposal: alternative psql commands quit and exit
Previous Message Simon Riggs 2018-02-01 13:16:05 Re: MCV lists for highly skewed distributions