Re: Parallel tuplesort (for parallel B-Tree index creation)

On Fri, Feb 3, 2017 at 4:15 PM, Thomas Munro
<thomas(dot)munro(at)enterprisedb(dot)com> wrote:
>> I suspect that this system isn't particularly well balanced for the
>> task of benchmarking the patch. You would probably see notably better
>> scalability than any you've shown in any test if you could add
>> additional sequential I/O bandwidth, which is probably an economical,
>> practical choice for many users. I suspect that you aren't actually
>> saturating available CPUs to the greatest extent that the
>> implementations makes possible.
>
> I will look into what IO options I can access before running larger
> tests. Also I will look into running the test with both cold and warm
> caches (ie "echo 1 > /proc/sys/vm/drop_caches") so that read bandwidth
> enters the picture.

It might just have been that the table was too small to be an
effective target for parallel sequential scan with so many workers,
and so a presorted best case CREATE INDEX, which isn't that different,
also fails to see much benefit (compared to what you'd see with a
similar case involving a larger table). In other words, I might have
jumped the gun in emphasizing issues with hardware and I/O bandwidth
over issues around data volume (that I/O parallelism is inherently not
very helpful with these relatively small tables).

As I've pointed out a couple of times before, bigger sorts will be
more CPU bound because sorting itself has costs that grow
linearithmically, whereas writing out runs has costs that grow
linearly. The relative cost of the I/O can be expected to go down as
input goes up for this reason. At the same time, a larger input might
make better use of I/O parallelism, which reduces the cost paid in
latency to write out runs in absolute terms.

--
Peter Geoghegan