From: | Andrew Jackson <andrewjackson947(at)gmail(dot)com> |
---|---|
To: | pgsql-hackers(at)lists(dot)postgresql(dot)org |
Cc: | Hannu Krosing <hannuk(at)google(dot)com>, Nitin Motiani <nitinmotiani(at)google(dot)com> |
Subject: | Re: Adding pg_dump flag for parallel export to pipes |
Date: | 2025-08-31 15:43:22 |
Message-ID: | 175665500252.297107.3384642490431951825.pgcf@coridan.postgresql.org |
Views: | Whole Thread | Raw Message | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Hi,
Went ahead and experimented with your patch a bit. To answer my previous question this patch can be used to pipe pg_dump directly into pg_restore. This should absolutely be added as another use case to your list above as it is a well known limitation that you can use pg_dump/psql to do buffered copy but only with a single process, while using pg_dump/pg_restore is capable of multiprocessed copy but it must be saved to disk in its entirety before the restore can begin. This is extremely frustrating when dealing with large databases where you don't want multiple copies saved on disk and because it's not as fast as it can be. With this patch you can get the best of both worlds.
Example dump
```bash
pg_dump --jobs=4 -Fd "${connection_str}" --pipe-command="mkfifo dumpdir/%f; cat >> dumpdir/%f"
```
Example restore run in different process
```bash
pg_restore --jobs=4 -Fd --dbname="${another_connection_str}" ./dumpdir
```
Thanks,
Andrew Jackson
From | Date | Subject | |
---|---|---|---|
Next Message | Arseniy Mukhin | 2025-08-31 16:17:20 | Use streaming read I/O in BRIN vacuuming |
Previous Message | Mihail Nikalayeu | 2025-08-31 15:29:00 | Re: Adding REPACK [concurrently] |