Re: WIP: WAL prefetch (another approach)

On Thu, Apr 8, 2021 at 3:27 AM Tomas Vondra
<tomas(dot)vondra(at)enterprisedb(dot)com> wrote:
> On 4/7/21 1:24 PM, Thomas Munro wrote:
> > I made one other simplifying change: previously, the prefetch module
> > would read the WAL up to the "written" LSN (so, allowing itself to
> > read data that had been written but not yet flushed to disk by the
> > walreceiver), though it still waited until a record's LSN was
> > "flushed" before replaying. That allowed prefetching to happen
> > concurrently with the WAL flush, which was nice, but it felt a little
> > too "special". I decided to remove that part for now, and I plan to
> > look into making standbys work more like primary servers, using WAL
> > buffers, the WAL writer and optionally the standard log-before-data
> > rule.
>
> Not sure, but the removal seems unnecessary. I'm worried that this will
> significantly reduce the amount of data that we'll be able to prefetch.
> How likely it is that we have data that is written but not flushed?
> Let's assume the replica is lagging and network bandwidth is not the
> bottleneck - how likely is this "has to be flushed" a limit for the
> prefetching?

Yeah, it would have been nice to include that but it'll have to be for
v15 due to lack of time to convince myself that it was correct. I do
intend to look into more concurrency of that kind for v15. I have
pushed these patches, updated to be disabled by default. I will look
into how I can run a BF animal that has it enabled during the recovery
tests for coverage. Thanks very much to everyone on this thread for
all the discussion and testing so far.