Re: [PoC] Improve dead tuple storage for lazy vacuum

On Wed, Mar 8, 2023 at 1:40 PM John Naylor <john(dot)naylor(at)enterprisedb(dot)com> wrote:
>
>

> On Tue, Mar 7, 2023 at 8:25 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> > > 1. Make it optional to track chunk memory space by a template parameter. It might be tiny compared to everything else that vacuum does. That would allow other users to avoid that overhead.
> > > 2. When context block usage exceeds the limit (rare), make the additional effort to get the precise usage -- I'm not sure such a top-down facility exists, and I'm not feeling well enough today to study this further.
> >
> > I prefer option (1) as it's straight forward. I mentioned a similar
> > idea before[1]. RT_MEMORY_USAGE() is defined only when the macro is
> > defined. It might be worth checking if there is visible overhead of
> > tracking chunk memory space. IIRC we've not evaluated it yet.
>
> Ok, let's try this -- I can test and profile later this week.

Thanks!

I've attached the new version patches. I merged improvements and fixes
I did in the v29 patch. 0007 through 0010 are updates from v29. The
main change made in v30 is to make the memory measurement and
RT_MEMORY_USAGE() optional, which is done in 0007 patch. The 0008 and
0009 patches are the updates for tidstore and the vacuum integration
patches. Here are results of quick tests (an average of 3 executions):

query: select * from bench_load_random_int(10 * 1000 * 1000)

* w/ RT_MEASURE_MEMORY_USAGE:
mem_allocated | load_ms
---------------+---------
1996512000 | 3305
(1 row)

* w/o RT_MEASURE_MEMORY_USAGE:
mem_allocated | load_ms
---------------+---------
0 | 3258
(1 row)

It seems to be within a noise level but I agree to make it optional.

Apart from the memory measurement stuff, I've done another todo item
on my list; adding min max classes for node3 and node125. I've done
that in 0010 patch, and here is a quick test result:

query: select * from bench_load_random_int(10 * 1000 * 1000)

* w/ 0000 patch
mem_allocated | load_ms
---------------+---------
1268630080 | 3275
(1 row)

* w/o 0000 patch
mem_allocated | load_ms
---------------+---------
1996512000 | 3214
(1 row)

That's a good improvement on the memory usage, without a noticeable
performance overhead. FYI CLASS_3_MIN has 1 fanout and is 24 bytes in
size, and CLASS_125_MIN has 61 fanouts and is 768 bytes in size.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com