Re: Tuplesort merge pre-reading

On Thu, Sep 29, 2016 at 2:59 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> Maybe that was the wrong choice of words. What I mean is that it seems
>> somewhat unprincipled to give over an equal share of memory to each
>> active-at-least-once tape, ...
>
> I don't get it. If the memory is being used for prereading, then the
> point is just to avoid doing many small I/Os instead of one big I/O,
> and that goal will be accomplished whether the memory is equally
> distributed or not; indeed, it's likely to be accomplished BETTER if
> the memory is equally distributed than if it isn't.

I think it could hurt performance if preloading loads runs on a tape
that won't be needed until some subsequent merge pass, in preference
to using that memory proportionately, giving more to larger input runs
for *each* merge pass (giving memory proportionate to the size of each
run to be merged from each tape). For tapes with a dummy run, the
appropriate amount of memory for there next merge pass is zero.

I'm not arguing that it would be worth it to do that, but I do think
that that's the sensible way of framing the idea of using a uniform
amount of memory to every maybe-active tape up front. I'm not too
concerned about this because I'm never too concerned about multiple
merge pass cases, which are relatively rare and relatively
unimportant. Let's just get our story straight.

--
Peter Geoghegan