Re: Adding pg_dump flag for parallel export to pipes

From: Andrew Jackson <andrewjackson947(at)gmail(dot)com>
To: pgsql-hackers(at)lists(dot)postgresql(dot)org
Cc: Hannu Krosing <hannuk(at)google(dot)com>, Nitin Motiani <nitinmotiani(at)google(dot)com>
Subject: Re: Adding pg_dump flag for parallel export to pipes
Date: 2025-08-31 15:43:22
Message-ID: 175665500252.297107.3384642490431951825.pgcf@coridan.postgresql.org
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

Went ahead and experimented with your patch a bit. To answer my previous question this patch can be used to pipe pg_dump directly into pg_restore. This should absolutely be added as another use case to your list above as it is a well known limitation that you can use pg_dump/psql to do buffered copy but only with a single process, while using pg_dump/pg_restore is capable of multiprocessed copy but it must be saved to disk in its entirety before the restore can begin. This is extremely frustrating when dealing with large databases where you don't want multiple copies saved on disk and because it's not as fast as it can be. With this patch you can get the best of both worlds.

Example dump
```bash
pg_dump --jobs=4 -Fd "${connection_str}" --pipe-command="mkfifo dumpdir/%f; cat >> dumpdir/%f"
```

Example restore run in different process
```bash
pg_restore --jobs=4 -Fd --dbname="${another_connection_str}" ./dumpdir
```
Thanks,
Andrew Jackson

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Arseniy Mukhin 2025-08-31 16:17:20 Use streaming read I/O in BRIN vacuuming
Previous Message Mihail Nikalayeu 2025-08-31 15:29:00 Re: Adding REPACK [concurrently]