Re: In-order pg_dump (or in-order COPY TO)

Sorry I was not remembering the details. Probably there is a TOC in your dump file, but it does not contain any positions for the data. The pg_restore command has to scan the whole file in advance, and fill in the TOC offsets in memory.

This scanning happens in a very inefficient way, with many seek calls and small block reads. Try strace to see them. This initial phase can take hours in a huge dump file, before even starting any actual restoration.

Thank you for testing.
Dimitris

On 30 August 2025 20:19:13 CEST, Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com> wrote:
>On 8/27/25 09:10, Dimitrios Apostolou wrote:
>>
>> On Wednesday 2025-08-27 17:25, Adrian Klaver wrote:
>
>
>>>
>>> For completeness and just in case they may affect the output what do the patches do?
>>
>> Two patches for speeding up scanning an archive without TOC, like the one I'm having (because it is piped into borg, instead of written to file). These were activated, but shouldn't matter. They only build the TOC in pg_restore's memory.
>
>Are you sure about that?
>
>I just did:
>
>pg_dump -Fc --compress=none --no-toast-compression -d test -U postgres | borg create --stats --stdin-name pg_file --stdin-user aklaver --stdin-group aklaver borg_test/::PgTest -
>
>Then:
>
>borg mount borg_test/ mnt_tmp/
>cd mnt_tmp/PgTest/
>
>and then:
>
>pg_restore -l pg_file
>
>and I got a TOC.
>
>Or are you streaming the data out of the Borg archive?
>
>>
>> https://commitfest.postgresql.org/patch/5809/
>> https://commitfest.postgresql.org/patch/5817/
>>
>> And two patches for speeding up pg_restore like mentioned above, under specific arguments that I didn't provide. (one speedup needs --clean, and the other needs --freeze).
>>
>> https://commitfest.postgresql.org/patch/5821/
>> https://commitfest.postgresql.org/patch/5826/
>>
>> IIRC I did not activate them (via --clean) because TRUNCATE fails when foreign keys exist. See the discussion threads.
>>
>>
>> Dimitris
>
>