Re: increasing the default WAL segment size

From: David Steele <david(at)pgmasters(dot)net>
To: Robert Haas <robertmhaas(at)gmail(dot)com>, Stephen Frost <sfrost(at)snowman(dot)net>, Beena Emerson <memissemerson(at)gmail(dot)com>, Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com>
Cc: tushar <tushar(dot)ahuja(at)enterprisedb(dot)com>, Prabhat Sahu <prabhat(dot)sahu(at)enterprisedb(dot)com>, Ashutosh Sharma <ashu(dot)coek88(at)gmail(dot)com>, Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com>, Kuntal Ghosh <kuntalghosh(dot)2007(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: increasing the default WAL segment size
Date: 2017-03-24 23:13:21
Message-ID: ff6d236f-7f91-38af-d86f-9c122fef9917@pgmasters.net
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Hi Robert,

On 3/24/17 3:00 PM, Robert Haas wrote:
> On Wed, Mar 22, 2017 at 6:05 PM, David Steele <david(at)pgmasters(dot)net> wrote:
>>> Wait, really? I thought you abandoned this approach because there's
>>> then no principled way to handle WAL segments of less than the default
>>> size.
>>
>> I did say that, but I thought I had hit on a compromise.
>>
>> But, as I originally pointed out the hex characters in the filename are not
>> aligned correctly for > 8 bits (< 16MB segments) and using different
>> alignments just made it less consistent.
>
> I don't think I understand what the compromise is. Are you saying we
> should have one rule for segments < 16MB and another rule for segments
>> 16MB? I think using two different rules for file naming depending
> on the segment size will be a negative for both tool authors and
> ordinary users.

Sorry for the confusion, I meant to say that if we want to keep LSNs in
the filenames and not change alignment for the current default, then we
would need to drop support for segment sizes < 16MB (more or less what I
said below). Bad editing on my part.

>> It would be OK if we were willing to drop the 1,2,4,8 segment sizes because
>> then the alignment would make sense and not change the current 16MB
>> sequence.
>
> Well, that is true. But the thing I'm trying to do here is to keep
> this patch down to what actually needs to be changed in order to
> accomplish the original purchase, not squeeze more and more changes
> into it.

Attached is a patch to be applied on top of Beena's v8 patch that
preserves LSNs in the file naming for all segment sizes. It's not quite
complete because it doesn't modify the lower size limit everywhere, but
I think it's enough so you can see what I'm getting at. This passes
check-world and I've poked at it in other segment sizes as well manually.

Behavior for the current default of 16MB is unchanged, and all other
sizes go through a logical progression.

1GB:
000000010000000000000040
000000010000000000000080
0000000100000000000000C0
000000010000000100000000

256GB:

000000010000000000000010
000000010000000000000020
000000010000000000000030
...
0000000100000000000000E0
0000000100000000000000F0
000000010000000100000000

64GB:

000000010000000100000004
000000010000000100000008
00000001000000010000000C
...
0000000100000001000000F8
0000000100000001000000FC
000000010000000100000000

I believe that maintaining an easy correspondence between LSN and
filename is important. The cluster will not always be up to help with
these calculations and tools that do the job may not be present or may
have issues.

I'm happy to merge this with Beena's patch (and tidy my patch up) if
this looks like an improvement to everyone.

--
-David
david(at)pgmasters(dot)net

Attachment Content-Type Size
wal_filename_lsn_match.patch text/plain 3.0 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Michael Paquier 2017-03-24 23:15:18 Re: Potential data loss of 2PC files
Previous Message Corey Huinker 2017-03-24 22:53:38 Re: \if, \elseif, \else, \endif (was Re: PSQL commands: \quit_if, \quit_unless)