RE: reducing memory usage by using "proxy" memory contexts?

From: "tsunakawa(dot)takay(at)fujitsu(dot)com" <tsunakawa(dot)takay(at)fujitsu(dot)com>
To: 'Andres Freund' <andres(at)anarazel(dot)de>
Cc: "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: RE: reducing memory usage by using "proxy" memory contexts?
Date: 2019-12-17 00:46:15
Message-ID: OSAPR01MB5073EFAEBB5C74A5F93F60E0FE500@OSAPR01MB5073.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

From: Andres Freund <andres(at)anarazel(dot)de>
> We waste a lot of space due to all these small contexts. Even leaving
> aside the overhead of the context and its blocks - not insignificant -
> they are mostly between ~1/2 a ~1/4 empty.
>
>
> But what if we had a new type of memory context that did not itself
> manage memory underlying allocations, but instead did so via the parent?
> If such a context tracked all the live allocations in some form of list,
> it could then free them from the parent at reset time. In other words,
> it'd proxy all memory management via the parent, only adding a separate
> name, and tracking of all live chunks.

It sounds like that it will alleviate the memory bloat caused by SAVEPOINT and RELEASE, which leave CurTransactionContext for each subtransaction. The memory overuse got Linux down when our customer's batch application ran millions of SQL statements in a transaction with psqlODBC. psqlODBC uses savepoints by default to enable statement rollback.

(I guess this issue of one memory context per subtransaction caused the crash of Amazon Aurora on the Prime Day last year.)

Regards
Takayuki Tsunakawa

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Stephen Frost 2019-12-17 00:46:32 Re: Unportable(?) use of setenv() in secure_open_gssapi()
Previous Message Michael Paquier 2019-12-17 00:36:13 Re: [Proposal] Level4 Warnings show many shadow vars