Re: BUG #15660: pg_dump memory leaks when dumping LOBs

Haribabu Kommi <kommi(dot)haribabu(at)gmail(dot)com> writes:
> On Wed, Feb 27, 2019 at 10:58 PM PG Bug reporting form <
> noreply(at)postgresql(dot)org> wrote:
>> One of our customers faced an unexpected behaviour when using pg_dump to
>> export one of DBs - pg_dump consumed all the server memory and got
>> terminated with following error:

> pg_dump process collects all the database objects first (tables, indexes
> and etc) and then it write them into the file/archive.

> In your scenario, there are many large objects that are present which leads
> to allocate memory for the each object before it gets dumped leads to out of
> memory.

Yeah. We've heard about this repeatedly since large objects were promoted
to be full-fledged objects with individual owners and ACL info. It was
argued at the time that large objects would be, well, large, and therefore
there wouldn't be *that* many of them, and so it'd be okay for pg_dump to
treat them as full-fledged objects with their own dump-table-of-contents
entries. Evidently, not everybody uses them that way :-(.

At some point I think we're going to have to revisit that decision in
pg_dump. Maybe we could merge things so that we have one TOC entry
for all blobs that share the same owner and ACL? But it comes up
seldom enough that nobody's gotten motivated to do the work.

In the short run, the only answer is to run pg_dump on beefier iron,
or else split up the dump as Haribabu suggests (although TBH I'm
afraid that might not work either --- if memory serves, pg_dump tends
to pull in all the per-object info even if it's not going to dump it
all ...)

regards, tom lane