From: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
---|---|
To: | Andrew Gierth <andrew(at)tao11(dot)riddles(dot)org(dot)uk>, "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com> |
Cc: | pgsql-hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: pg_dump: largeobject behavior issues (possible bug) |
Date: | 2015-04-23 21:08:31 |
Message-ID: | 55395F4F.3090101@dunslane.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On 04/23/2015 04:04 PM, Andrew Gierth wrote:
>>>>>> "Joshua" == Joshua D Drake <jd(at)commandprompt(dot)com> writes:
> Joshua> The database dumps fine as long as we don't dump large
> Joshua> objects. However, if we try to dump the large objects, FreeBSD
> Joshua> will kill pg_dump as it will consume all free memory and
> Joshua> swap. With Andrew's help we were able to determine the
> Joshua> following:
>
> Joshua> There is a memory cost of about 160 bytes per largeobject.
>
> I may have the exact number here wrong, it was just a quick eyeball of
> the data structures (and depends on malloc overheads anyway).
>
> The relevant code is getBlobs in pg_dump.c, which queries the whole of
> pg_largeobject_metadata without using a cursor (so the PGresult is
> already huge thanks to having >100 million rows), and then mallocs a
> BlobInfo array and populates it from the PGresult, also using pg_strdup
> for the oid string, owner name, and ACL if any.
>
I'm surprised this hasn't come up before. I have a client that I
persuaded to convert all their LOs to bytea fields because of problems
with pg_dump handling millions of LOs, and kept them on an older
postgres version until they made that change.
cheers
andrew
From | Date | Subject | |
---|---|---|---|
Next Message | Peter Geoghegan | 2015-04-23 23:52:39 | Re: INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0 |
Previous Message | David Steele | 2015-04-23 20:42:23 | Re: tablespaces inside $PGDATA considered harmful |