Re: Patch: dumping tables data in multiple chunks in pg_dump

Here is a new patch which has

* changed flag name to max-table-segment-pages
* added check for amname = "heap"
* made the table info query use pg_relation_size() to get relpages if
the --max-table-segment-pages is set
* added simple chunked dump and restore test

Currently there is no check for actual restore integrity, this is
what t/002_pg_dump.pl says:

# TODO: Have pg_restore actually restore to an independent
# database and then pg_dump *that* database (or something along
# those lines) to validate that part of the process.

As my perl-fu is weak I did not build the new facility to have full
restored data checking, but I did add simple count + table hash
warnings for original and restored data so I could manually verify tha
restore

added this for original and chunked restore database:

DO \$\$
DECLARE
thash_rec RECORD;
BEGIN
SELECT 'tplain', count(*), sum(hashtext(t::text)) as tablehash
INTO thash_rec
FROM tplain AS t;
RAISE WARNING 'thash after parallel chunked restore: %', thash_rec;
END;
\$\$;

And this is the verification I did after running `make check` in
src/bin/pg_dump/

hannu(at)HK395:~/work/pggit/src/bin/pg_dump$ grep "WARNING.*thash"
tmp_check/log/004_pg_dump_parallel_main.log
RAISE WARNING 'thash: %', thash_rec;
2026-01-19 19:27:57.444 CET client backend[678937]
004_pg_dump_parallel.pl WARNING: thash: (tplain,1000,38441792160)
RAISE WARNING 'thash after parallel chunked restore: %', thash_rec;
2026-01-19 19:27:57.605 CET client backend[678985]
004_pg_dump_parallel.pl WARNING: thash after parallel chunked
restore: (tplain,1000,38441792160)

As you see both have 1000 rows with sum of full row hashes == 38441792160

Other rows in the same log foile show that it was dumped as 3 chunks
as I still have the Warnings in code which show the query used.

Anyone with a better understanding of our Perl tests is welcome to
turn this into proper tests or advise me where to find info on how to
do it.

On Tue, Jan 13, 2026 at 3:27 AM David Rowley <dgrowleyml(at)gmail(dot)com> wrote:
>
...
> 3. You should be able to simplify the code in dumpTableData() so
> you're not adding any extra cases. You could use InvalidBlockNumber to
> indicate an unbounded ctid range and only add ctid qual to the WHERE
> clause when you have a bounded range (i.e not InvalidBlockNumber).
> That way the first segment will need WHERE ctid <= '...' and the final
> one will need WHERE ctid >= '...'. Everything in between will have an
> upper and lower bound. That results in no ctid quals being added when
> both ranges are set to InvalidBlockNumber, which you should use for
> all tables not large enough to be segmented, thus no special case.
>
> TID Range scans are perfectly capable of working when only bounded at one side.

I changed the last open-ended chunk to use ctid >= (N,1) for clarity
but did not change anything else.

To me it looked like having a loop around the whole thing when there
is no chunking would complicate things for anyone reading the code.

> 4. I think using "int" here is a future complaint waiting to happen.
>
> + if (!option_parse_int(optarg, "--huge-table-chunk-pages", 1, INT32_MAX,
> + &dopt.huge_table_chunk_pages))
>
> I bet we'll eventually see a complaint that someone can't make the
> segment size larger than 16TB. I think option_parse_uint32() might be
> called for.

I have not yet done anything with this yet, so the maximum chunk size
for now is half of the maximum relpages.