| From: | Stepan Neretin <slpmcf(at)gmail(dot)com> |
|---|---|
| To: | PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Subject: | [PATCH 0/2] Parallelizing the archiver |
| Date: | 2026-11-22 18:16:33 |
| Message-ID: | CA+Yyo5TfMpUE3s=WKrWcaYbBWbO2VNyRnRt_5hy-VPrMhjXVWg@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
Follow up to [1] and the older parallel archiver threads [2][3].
Today archiving often runs in peaks. One file starts, finishes, then the
next starts, so throughput is stop and go. That is especially noticeable
when the archive target is slow, for example object storage such as S3 or
a distant NFS share, where each file pays a long wait. Async work keeps
several files in flight and reports each one as it completes, so
utilization stays high instead of sitting idle between handoffs.
There is a second pain point. pgarch currently blocks inside system() for
the whole archive_command, so a hung script can leave the archiver stuck
until someone kills the child by hand.
The idea here is simple. pgarch keeps a small fixed pool of archive
workers. Ready files go into free slots, and when a worker finishes the
next file takes its place. Core still marks .done and can cancel stuck
work on shutdown. archive_command_jobs sets the pool size, default 1.
On Unix values from 2 to 8 start one short lived archive_command child
per slot. The feature is not available on Windows yet. Existing sync
modules keep working as before.
Attached archive_jobs_bench.sh builds a .ready backlog, then times how
long archive_command_jobs takes to drain it. The command optionally
sleeps, then copies each WAL file into an archive directory.
PGBIN=/path/to/bin ./archive_jobs_bench.sh [rounds] [delay_ms/jobs ...]
delay=500ms + cp, 16 files (median of 3)
jobs=1 8.35s
jobs=2 4.18s (~2.0x)
jobs=4 2.12s (~3.9x)
jobs=8 1.10s (~7.6x)
delay=0 + cp, 120 files (median of 3)
jobs=1 1.93s
jobs=2 1.02s (~1.9x)
jobs=4 0.43s (~4.5x)
jobs=8 0.29s (~6.7x)
Andrey Borodin put the same peak load complaint from the walg side and
preferred a pipe style protocol over batches. Would it make sense to grow
this later into that shape, with pgarch feeding file names and getting
completed or failed files back one by one?
I used AI to sketch the first prototype, then cleaned it up by hand. The
series is still large, but the shape looks promising to me. Does this
direction seem reasonable?
Regards,
Stepan Neretin
[1]
https://www.postgresql.org/message-id/CA%2BYyo5QRxow54ekSUWu4qzckYWhSg5BWiFfQ37UdU3QGBC3Kxg%40mail.gmail.com
[2]
https://www.postgresql.org/message-id/flat/BC335D75-105B-403F-9473-976C8BBC32E3%40yandex-team.ru
[3]
https://www.postgresql.org/message-id/BC4D6BB2-6976-4397-A417-A6A30EEDC63E%40amazon.com
| Attachment | Content-Type | Size |
|---|---|---|
| 0001-Support-async-archive-callbacks-in-pgarch.patch | text/x-patch | 23.9 KB |
| 0002-Add-archive_command_jobs.patch | text/x-patch | 22.9 KB |
| archive_jobs_bench.sh | text/x-shellscript | 4.3 KB |
| From | Date | Subject | |
|---|---|---|---|
| Previous Message | Zhijie Hou | 2026-11-22 17:46:12 | Fix LogStandbySnapshot() race during logical decoding activation |