Re: [Patch] Make block and file size for WAL and relations defined at cluster creation

Robert,

Justifications are:

- we may test different combinations of file and block sizes, for the
relation and the WAL in order to have the better performances of the server.
Avoiding a compilation for each combination of values seems to make sense.

- the same binary can be used on the same host with several databases
instances/cluster, each using different values for block and file sizes.
This is what I did to test the patch. I have created about 20 different
combinations of values for the file and block sizes of the relation and WAL
files.

- Linux distributions deliver Postgresql with a binary already compiled
with the default values.
This means DBAs need to rebuild the binary for each combination of block
and file sizes, whether this is for the WAL or the relations.

- Selecting the correct values for file and block sizes is a DBA task, and
not a developer task.
For instance, when someone wants to create a Linux filesystem with a given
block size, he is not forced to accept a given value chosed by the
developer of the filesystem driver when this later was compiled.

- The file and block sizes should depend mostly of the physical server and
physical storage.
Not of the database software itself.

Regarding the cost of using run-time configurable values for file and block
sizes of the WAL and relations, this cost is low both :

- from a developer point of view: the source code changes are spread in
many files but only a few one have significant changes.
Mainly the tidbitmap.c is concerned the change. Other changes are minor
changes.

- from a run-time point of view. The overhead is only at the start of the
database instance.
And moreover, the overhead is still very low at the start of the server,
with only a few more dynamic memory allocations.

Regards
Remi

2018-01-02 22:26 GMT+01:00 Robert Haas <robertmhaas(at)gmail(dot)com>:

> On Sun, Dec 31, 2017 at 12:00 PM, Remi Colinet <remi(dot)colinet(at)gmail(dot)com>
> wrote:
> > Below patch makes block and file sizes defined at cluster creation for
> both
> > the WAL and the relations. This avoids having different server builds for
> > each possible combination of block size and file sizes.\
>
> The email thread where we discussed making the WAL segment size
> configurable at initdb time contained a detailed rationale, explaining
> why it was useful to be able to make such a change. The very short
> version is that, if a system is generating WAL at a very high rate,
> being able to group that WAL into fewer, larger files makes life
> easier since, for example, the latency requirements for
> archive_command are not as tight, and "ls pg_wal" doesn't have to go
> into the tank just trying to read the directory contents.
>
> Your email doesn't seem to contain a rationale explaining why the
> block and file sizes should be run-time configurable. There may be a
> very good reason, but can you explain what it is?
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>