Re: Storage Model for Partitioning

On Fri, 2008-01-11 at 11:34 +0000, Richard Huxton wrote:

> Is the following basically the same as option #3 (multiple RelFileNodes)?
>
> 1. Make an on-disk "chunk" much smaller (e.g. 64MB). Each chunk is a
> contigous range of blocks.
> 2. Make a table-partition (implied or explicit constraints) map to
> multiple "chunks".
> That would reduce fragmentation (you'd have on average 32MB's worth of
> blocks wasted per partition) and allow for stretchy partitions at the
> cost of an extra layer of indirection.
>
> For the single-partition case you'd not need to split the file of
> course, so it would end up looking much like the current arrangement.

We need to think about the "data model" of the storage layer. Space
itself isn't the issue, its the assumptions that all of the other
subsystems currently make about what how a table is structured, indexed,
accessed and manipulated.

Currently: Table 1:M Segments

Option 1: Table 1:M Segments and *separately* Table 1:M Partitions, so
partitions are always have a maximum size. The size just changes the
impact, doesn't change the impact of holes, max sizes etc.
e.g. empty table with 10 partitions would be
a) 0 bytes in 1 file
b) 0 bytes in 1 file, plus 9GB in 9 files all full of empty blocks

e.g. table with 10 partitions each of 1.5GB would be
a) 15 GB in 15 files
b) hit max size limit of partition: ERROR

Option 2: Table 1:M Child Tables 1:M Segments
e.g. empty table with 10 partitions would be
0 bytes in each of 10 files

e.g. table with 10 partitions each of 1.5GB would be
15GB in 10 groups of 2 files

Option 3: Table 1:M Nodes 1:M Segments
e.g. empty table with 10 partitions would be
0 bytes in each of 10 files

e.g. table with 10 partitions each of 1.5GB would be
15GB in 10 groups of 2 files

So 1b) seems definitely out.

The implications of 2 and 3 are what I'm worried about, which is why the
shortcomings of 1a) seem acceptable currently.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com