Re: Option to not use ringbuffer in VACUUM, using it in failsafe mode

On Tue, Mar 21, 2023 at 6:03 AM Ants Aasma <ants(at)cybertec(dot)at> wrote:
>
> On Mon, 20 Mar 2023 at 00:59, Melanie Plageman
> <melanieplageman(at)gmail(dot)com> wrote:
> >
> > On Wed, Mar 15, 2023 at 6:46 AM Ants Aasma <ants(at)cybertec(dot)at> wrote:
> > >
> > > On Wed, 15 Mar 2023 at 02:29, Melanie Plageman
> > > <melanieplageman(at)gmail(dot)com> wrote:
> > > > As for routine vacuuming and the other buffer access strategies, I think
> > > > there is an argument for configurability based on operator knowledge --
> > > > perhaps your workload will use the data you are COPYing as soon as the
> > > > COPY finishes, so you might as well disable a buffer access strategy or
> > > > use a larger fraction of shared buffers. Also, the ring sizes were
> > > > selected sixteen years ago and average server memory and data set sizes
> > > > have changed.
> > >
> > > To be clear I'm not at all arguing against configurability. I was
> > > thinking that dynamic use could make the configuration simpler by self
> > > tuning to use no more buffers than is useful.
> >
> > Yes, but I am struggling with how we would define "useful".
>
> For copy and vacuum, the only reason I can see for keeping visited
> buffers around is to avoid flushing WAL or at least doing it in larger
> batches. Once the ring is big enough that WAL doesn't need to be
> flushed on eviction, making it bigger only wastes space that could be
> used by something that is not going to be evicted soon.

Well, I think if you know you will use the data you are COPYing right
away in your normal workload, it could be useful to have the ring be
large or to disable use of the ring. And, for vacuum, if you need to get
it done as quickly as possible, again, it could be useful to have the
ring be large or disable use of the ring.

> > > > StrategyRejectBuffer() will allow bulkreads to, as you say, use more
> > > > buffers than the original ring size, since it allows them to kick
> > > > dirty buffers out of the ring and claim new shared buffers.
> > > >
> > > > Bulkwrites and vacuums, however, will inevitably dirty buffers and
> > > > require flushing the buffer (and thus flushing the associated WAL) when
> > > > reusing them. Bulkwrites and vacuum do not kick dirtied buffers out of
> > > > the ring, since dirtying buffers is their common case. A dynamic
> > > > resizing like the one you suggest would likely devolve to vacuum and
> > > > bulkwrite strategies always using the max size.
> > >
> > > I think it should self stabilize around the point where the WAL is
> > > either flushed by other commit activity, WAL writer or WAL buffers
> > > filling up. Writing out their own dirtied buffers will still happen,
> > > just the associated WAL flushes will be in larger chunks and possibly
> > > done by other processes.
> >
> > They will have to write out any WAL associated with modifications to the
> > dirty buffer before flushing it, so I'm not sure I understand how this
> > would work.
>
> By the time the dirty buffer needs eviction the WAL associated with it
> can already be written out by concurrent commits, WAL writer or by WAL
> buffers filling up. The bigger the ring is, the higher the chance that
> one of these will happen before we loop around.

Ah, I think I understand the idea now. So, I think it is an interesting
idea to try and find the goldilocks size for the ring buffer. It is
especially interesting to me in the case in which we are enlarging the
ring.

However, given that concurrent workload variability, machine I/O latency
fluctuations, etc, we will definitely have to set a max value of some
kind anyway for the ring size. So, this seems more like a complimentary
feature to vacuum_buffer_usage_limit. If we added some kind of adaptive
sizing in a later version, we could emphasize in the guidance for
setting vacuum_buffer_usage_limit that it is the *maximum* size you
would like to allow vacuum to use. And, of course, there are the other
operations which use buffer access strategies.

> > > > As for decreasing the ring size, buffers are only "added" to the ring
> > > > lazily and, technically, as it is now, buffers which have been added
> > > > added to the ring can always be reclaimed by the clocksweep (as long as
> > > > they are not pinned). The buffer access strategy is more of a
> > > > self-imposed restriction than it is a reservation. Since the ring is
> > > > small and the buffers are being frequently reused, odds are the usage
> > > > count will be 1 and we will be the one who set it to 1, but there is no
> > > > guarantee. If, when attempting to reuse the buffer, its usage count is
> > > > > 1 (or it is pinned), we also will kick it out of the ring and go look
> > > > for a replacement buffer.
> > >
> > > Right, but while the buffer is actively used by the ring it is
> > > unlikely that clocksweep will find it at usage 0 as the ring buffer
> > > should cycle more often than the clocksweep. Whereas if the ring stops
> > > using a buffer, clocksweep will eventually come and reclaim it. And if
> > > the ring shrinking decision turns out to be wrong before the
> > > clocksweep gets around to reusing it, we can bring the same buffer
> > > back into the ring.
> >
> > I can see what you mean about excluding a buffer from the ring being a
> > more effective way of allowing it to be reclaimed. However, I'm not sure
> > I understand the use case. If the operation, say vacuum, is actively
> > using the buffer and keeping its usage count at one, then what would be
> > the criteria for it to decide to stop using it?
>
> The criteria for reducing ring size could be that we have cycled the
> ring buffer n times without having to do any WAL flushes.
>
> > Also, if vacuum used the buffer once and then didn't reuse it but, for
> > some reason, the vacuum isn't over, it isn't any different at that point
> > than some other buffer with a usage count of one. It isn't any harder
> > for it to be reclaimed by the clocksweep.
> >
> > The argument I could see for decreasing the size even when the buffers
> > are being used by the operation employing the strategy is if there is
> > pressure from other workloads to use those buffers. But, designing a
> > system that would reclaim buffers when needed by other workloads is more
> > complicated than what is being proposed here.
>
> I don't think any specific reclaim is needed, if the ring stops using
> a buffer *and* there is pressure from other workloads the buffer will
> get used for other stuff by the normal clocksweep. If the ring keeps
> using it then the normal clocksweep is highly unlikely to find it with
> usage count 0. If there is no concurrent allocation pressure, the ring
> can start using it again if that turns out to be necessary (probably
> should still check that it hasn't been reused by someone else).

Yes, you don't need a specific reclaim mechanism. But you would want to
be quite conservative about decreasing the ring size (given workload
variation and machine variations such as bursting in the cloud) and
probably not do so simply because the operation using the strategy
doesn't absolutely need the buffer but also because other concurrent
workloads really need the buffer. And, it seems complicated to determine
if other workloads do need the buffer.

- Melanie