Re: Adding pg_dump flag for parallel export to pipes

From: Nitin Motiani <nitinmotiani(at)google(dot)com>
To: Hannu Krosing <hannuk(at)google(dot)com>
Cc: Mahendra Singh Thalor <mahi6run(at)gmail(dot)com>, Dilip Kumar <dilipbalaut(at)gmail(dot)com>, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Adding pg_dump flag for parallel export to pipes
Date: 2026-05-04 04:39:13
Message-ID: CAH5HC97skKgTkOBUipZjpOCXYc27txdzFL8UmVmBKaRvc4LO5A@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Mahendra,

I'm attaching the latest version of the patch. This incorporates
several suggestions Mahendra made. I've also fixed the failing test
and added support for Windows in the test by using perlbin instead of
cat. Here is the list of changes in this version:

1. Changed pipe-command to pip.
2. Added better error handling for invalid command.
3. Added shell escaping in the command before setting it as the file path.
4. I decided to change the mode to PG_BINARY_W for the large object
toc files even in the standard case. If there is a concern, we can
revert to the older version.
5. Added a bunch of new tests for various scenarios and parallelism.

Currently these changes are in a separate commit. After the review, I
can squash the first 3 commits of pg_dump, pg_restore, and the fixes
in one commit.

On Sat, Mar 14, 2026 at 10:37 PM Mahendra Singh Thalor
<mahi6run(at)gmail(dot)com> wrote:
>
> Comment5: I think we can support this new pipe option with pg_dumpall also as we support directory mode in pg_dumpall from v19.

Regarding this, I haven't looked into the details of how that would
work. If the rest look good, we can consider implementing this in
another patch. For the time being, I've skipped this option in
pg_dumpall and the global restore. Let me know what you think.

Thanks

Nitin Motiani,
Google

Attachment Content-Type Size
v11-0003-Fixes-and-refactors-in-pipe-command.patch application/x-patch 40.2 KB
v11-0004-Add-tests-for-pipe.patch application/x-patch 14.9 KB
v11-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch application/x-patch 7.4 KB
v11-0002-Add-pipe-command-support-in-pg_restore.patch application/x-patch 9.5 KB
v11-0001-Add-pipe-command-support-for-directory-mode-of-p.patch application/x-patch 31.8 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Previous Message Tom Lane 2026-05-04 04:38:38 Re: Make printtup a bit faster