| From: | Japin Li <japinli(at)hotmail(dot)com> |
|---|---|
| To: | Pg Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Subject: | Support long tar paths and link targets using PAX |
| Date: | 2026-11-07 12:40:36 |
| Message-ID: | SY7PR01MB10921430D7F1623CF0B45C985B6752@SY7PR01MB10921.ausprd01.prod.outlook.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi hackers,
Currently, pg_basebackup can fail when creating plain-format backups if
a file path or symbolic link target cannot be represented in a
traditional ustar header.
The ustar format has fixed-size fields for member names and symbolic link
targets. Although long member names can be split between the `name` and
`prefix` fields, some paths still cannot be represented. Symbolic link
targets, on the other hand, are limited to 100 bytes.
This can be problematic when backing up tablespaces located in
directories with long paths, resulting in errors such as:
ERROR: symbolic link target too long for tar format:
file name "pg_tblspc/16386", target ".../007_tablespace/data/tblspc_djiS"
This patch adds support for POSIX PAX extended headers to handle these cases.
The implementation has the following changes:
- Add `tarCreateHeaderExt()` and `tarCreateHeaderExtSize()` to generate
tar headers with optional PAX extended headers.
- Prefer the traditional ustar representation whenever possible,
including splitting member names between the `name` and `prefix`
fields, so existing archives remain unchanged when their paths fit.
- Use the PAX `path` and `linkpath` attributes when member names or
symbolic link targets cannot be represented in a ustar header.
- Teach the astreamer tar parser to consume local PAX extended headers
and apply their attributes to the following archive member.
- Reject global PAX extended headers and ignore unknown attributes that
do not affect the properties handled by the parser.
- Update the tar archiver to reconstruct headers when PAX attributes need
to be preserved during re-archiving.
- Update the file extractor and pg_verifybackup to handle longer member
paths.
TAP tests are added to cover:
- Backups containing long file paths.
- Tablespaces with long paths and symbolic link targets.
- Verification of PAX-format backups using pg_verifybackup.
- Extraction of generated archives using an external tar utility.
- Re-archiving when pg_basebackup writes tar output to stdout.
Comments and reviews are welcome.
--
Regards,
Japin Li
ChengDu WenWu Information Technology Co., Ltd.
| Attachment | Content-Type | Size |
|---|---|---|
| v1-0001-Support-long-tar-paths-and-link-targets-using-PAX.patch | text/x-patch | 53.0 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Xuneng Zhou | 2026-11-07 13:05:56 | Re: BGWORKER_INTERRUPTIBLE workers are never restarted after DROP/ALTER/CREATE DATABASE |
| Previous Message | Bertrand Drouvot | 2026-11-07 09:50:29 | Re: Draft back-branch release notes are up |