Re: Freeze avoidance of very large table.

From: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com>, Peter Geoghegan <pg(at)heroku(dot)com>, Bruce Momjian <bruce(at)momjian(dot)us>, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>, Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>, Michael Paquier <michael(dot)paquier(at)gmail(dot)com>, Jeff Janes <jeff(dot)janes(at)gmail(dot)com>, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, Simon Riggs <simon(at)2ndquadrant(dot)com>, Josh Berkus <josh(at)agliodbs(dot)com>, Fujii Masao <masao(dot)fujii(at)gmail(dot)com>, Petr Jelinek <petr(at)2ndquadrant(dot)com>, Greg S <stark(at)mit(dot)edu>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Freeze avoidance of very large table.
Date: 2016-03-07 17:41:57
Message-ID: CAD21AoA73jervw5P0wsRF8eaMS+Kek4db-p5OGHxfn6D4xQMWA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Sat, Mar 5, 2016 at 11:25 PM, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> On Sat, Mar 5, 2016 at 1:25 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Wed, Mar 2, 2016 at 6:41 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> writes:
>>>> On 3/2/16 4:21 PM, Peter Geoghegan wrote:
>>>>> I think you should commit this. The chances of anyone other than you
>>>>> and Masahiko recalling that you developed this tool in 3 years is
>>>>> essentially nil. I think that the cost of committing a developer-level
>>>>> debugging tool like this is very low. Modules like pg_freespacemap
>>>>> currently already have no chance of being of use to ordinary users.
>>>>> All you need to do is restrict the functions to throw an error when
>>>>> called by non-superusers, out of caution.
>>>>>
>>>>> It's a problem that modules like pg_stat_statements and
>>>>> pg_freespacemap are currently lumped together in the documentation,
>>>>> but we all know that.
>>>
>>>> +1.
>>>
>>> Would it make any sense to stick it under src/test/modules/ instead of
>>> contrib/ ? That would help make it clear that it's a debugging tool
>>> and not something we expect end users to use.
>>
>> I actually think end-users might well want to use it. Also, I created
>> it by hacking up pg_freespacemap, so it may make sense to have it in
>> the same place.
>> I would also be tempted to add an additional C functions that scan the
>> entire visibility map and return counts of the total number of bits of
>> each type that are set, and similarly for the page level bits.
>> Presumably that would be much faster than
>
> +1.
>
>>
>> I am also tempted to change the API to be a bit more friendly,
>> although I am not sure exactly how. This was a quick and dirty hack
>> so that I could test, but the hardest thing about making it not a
>> quick and dirty hack is probably deciding on a good UI.
>>
>
> Does it mean visibility map API in visibilitymap.c?
>

Attached latest version optimisation patch.
I'm still consider regarding pg_upgrade regression test code, so I
will submit that patch later.

Regards,

--
Masahiko Sawada

Attachment Content-Type Size
000_optimize_vacuum_using_freezemap_v37.patch application/octet-stream 24.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2016-03-07 17:50:33 Re: checkpointer continuous flushing - V16
Previous Message Andres Freund 2016-03-07 17:41:51 Re: checkpointer continuous flushing - V16