From: | Craig Ringer <craig(at)2ndquadrant(dot)com> |
---|---|
To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
Cc: | Justin Pryzby <pryzby(at)telsasoft(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: [GENERAL] huge RAM use in multi-command ALTER of table heirarchy |
Date: | 2017-07-19 01:47:07 |
Message-ID: | CAMsr+YFJa24FGiU1_6Z8wygd-jmvQ7OF1bJk0i8U18wSz2+QXQ@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general pgsql-hackers |
On 19 July 2017 at 07:26, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Justin Pryzby <pryzby(at)telsasoft(dot)com> writes:
> > I've seen this before while doing SET STATISTICS on a larger number of
> columns
> > using xargs, but just came up while doing ADD of a large number of
> columns.
> > Seems to be roughly linear in number of children but superlinear WRT
> columns.
> > I think having to do with catalog update / cache invalidation with many
> > ALTERs*children*columns?
>
> I poked into this a bit. The operation is necessarily roughly O(N^2) in
> the number of columns, because we rebuild the affected table's relcache
> entry after each elementary ADD COLUMN operation, and one of the principal
> components of that cost is reading all the pg_attribute entries. However,
> that should only be a time cost not a space cost. Eventually I traced the
> O(N^2) space consumption to RememberToFreeTupleDescAtEOX, which seems to
> have been introduced in Simon's commit e5550d5fe, and which strikes me as
> a kluge of the first magnitude. Unless I am missing something, that
> function's design concept can fairly be characterized as "let's leak
> memory like there's no tomorrow, on the off chance that somebody somewhere
> is ignoring basic coding rules".
>
> I tried ripping that out, and the peak space consumption of your example
> (with 20 child tables and 1600 columns) decreased from ~3GB to ~200MB.
> Moreover, the system still passes make check-world, so it's not clear
> to me what excuse this code has to live.
>
> It's probably a bit late in the v10 cycle to be taking any risks in
> this area, but I'd vote for ripping out RememberToFreeTupleDescAtEOX
> as soon as the v11 cycle opens, unless someone can show an example
> of non-broken coding that requires it. (And if so, there ought to
> be a regression test incorporating that.)
Just FYI, I believe Simon's currently on holiday, so may not notice this
discussion as promptly as usual.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
From | Date | Subject | |
---|---|---|---|
Next Message | Patrick B | 2017-07-19 01:56:41 | PG 9.1 - FK + Check constraint |
Previous Message | Tom Lane | 2017-07-18 23:26:30 | Re: [GENERAL] huge RAM use in multi-command ALTER of table heirarchy |
From | Date | Subject | |
---|---|---|---|
Next Message | Mark Dilger | 2017-07-19 04:13:10 | Re: Something for the TODO list: deprecating abstime and friends |
Previous Message | Amit Langote | 2017-07-19 01:30:42 | Re: Oddity in error handling of constraint violation in ExecConstraints for partitioned tables |