Re: Adding pg_dump flag for parallel export to pipes

From: Nitin Motiani <nitinmotiani(at)google(dot)com>
To: solai v <solai(dot)cdac(at)gmail(dot)com>
Cc: Hannu Krosing <hannuk(at)google(dot)com>, Mahendra Singh Thalor <mahi6run(at)gmail(dot)com>, Dilip Kumar <dilipbalaut(at)gmail(dot)com>, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Adding pg_dump flag for parallel export to pipes
Date: 2026-08-08 10:25:00
Message-ID: CAH5HC94uWqwGwWzCYHB_h4-jt1dT_M1=th+3mJaxnAdZ5XBAMg@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri, Jun 12, 2026 at 10:50 AM solai v <solai(dot)cdac(at)gmail(dot)com> wrote:
>

> without creating an intermediate dump directory.
> I verified error handling (--pipe='invalid_cmd') as well and correctly
> reported the error as "pg_dump: error: pipe command failed:
> "invalid_cmd": command not found".
> Similarly, --pipe='gzip | false' correctly propagated the child
> process failure. I also performed an end-to-end validation of the new
> functionality by restoring the generated archive using pg_restore
> --pipe. The restore completed successfully, and row counts for the
> test tables matched the original database confirming that dump
> generation, compression, restore, and data integrity all worked
> correctly. One particularly useful improvement I noticed is the
> handling of parallel jobs when the %f placeholder is omitted,
> producing the warning: "pg_dump: warning: parallel jobs with --pipe
> usually require the "%f" placeholder to avoid data corruption from
> multiple workers writing to the same file". This is an usability
> improvement, as it explicitly warns users against a common misuse that
> could otherwise lead to corrupted output when multiple workers write
> to the same destination file. Overall, the patch series worked well
> and the new --pipe support behaves correctly in my testing, also the
> newly added warning for missing %f significantly improves the user
> experience for parallel jobs. The patch looks good to me.
>
>

Thanks for the feedback. Rebasing and uploading v18.

Attachment Content-Type Size
v18-0003-Fixes-and-refactors-in-pipe-command.patch application/x-patch 35.9 KB
v18-0002-Add-pipe-command-support-in-pg_restore.patch application/x-patch 6.0 KB
v18-0004-Add-tests-for-pipe.patch application/x-patch 22.1 KB
v18-0001-Add-pipe-command-support-for-directory-mode-of-p.patch application/x-patch 30.6 KB
v18-0005-Add-documentation-for-pipe-in-pg_dump-and-pg_res.patch application/x-patch 7.3 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Previous Message Henri GASC 2026-08-08 09:53:21 [SQL/PGQ] Native executor for Graph query