Re: Proposal: SLRU to Buffer Cache

On Tue, Aug 21, 2018 at 07:15:28AM -0700, Andres Freund wrote:

> On 2018-08-21 09:53:21 -0400, Shawn Debnath wrote:
> > > I was wondering what the point of exposing the OIDs to users in a
> > > catalog would be though. It's not necessary to do that to reserve
> > > them (and even if it were, pg_database would be the place): the OIDs
> > > we choose for undo, clog, ... just have to be in the system reserved
> > > range to be safe from collisions.
>
> Maybe I'm missing something, but how are conflicts prevented just by
> being in the system range? There's very commonly multiple patches
> trying to use the same oid, and that is just discovered by the
> 'duplicate_oids' script. But if there's no catalog representation, I
> don't see how that'd discover them?

+1. That's the reason why I suggested introducing a new catalog to
reserve, and at the same, expose the OIDs to the customer. To Thomas'
point, we can worry about the catalog after the storage manager patch is
ready for review as the values will need to be hard coded anyways to
avoid bootstrap issues. More on the catalog below.

> > > I suppose one benefit would be the
> > > ability to join eg pg_buffer_cache against it to get a human readable
> > > name like "clog", but that'd be slightly odd because the DB OID field
> > > would refer to entries in pg_database or pg_storage_manager depending
> > > on the number range.
>
> > Good points. However, there are very few cases where our internal
> > representation using DB OIDs will be exposed, one such being
> > pg_buffercache. Wondering if updating the documentation here would be
> > sufficient as pg_buffercache is an extension used by developers and DBEs
> > rather than by consumers. We can circle back to this after the initial
> > set of patches are out.
>
> Showing the oids in pg_database or such seems like it'd make it a bit
> harder to change later because people rely on things like joining
> against it. I don't think I like that. I'm kinda inclined to something
> somewhat crazy like instead having a reserved & shared pg_class entry or
> such. Don't like that that much either. Hm.

For some scenarios, like SLRU, we could follow the scheme that is used
by pg_database and have it be shared across all databases. Entries could
be inserted into pg_class and we can introduce a new ‘reserved’ or
‘system’ relkind for it. The files would reside in the global
tablespace.

Unfortunately, it wouldn’t be a good fit for undo logs as it uses the
relation OID for the dynamic set of undo logs under one specific DB OID.
Because of this, I was taking the DB OID approach but instead of adding
it to pg_database which is inflexible with its member types, I suggested
that we introduce a new catalog to track the OIDs. Any reason why we
shouldn’t/can’t introduce a new catalog to track these?

If a new catalog is not possible, I would prefer the pg_class approach
given we can define a new relkind to track these special relations and
have the rest of the fields be set to invalid state if needed (which is
supported today). This would require significant re-work for the undo
log implementation.

> > > > 5. Due to the on-disk format changes, simply copying the segments
> > > > during upgrade wouldn't work anymore. Given the nature of data
> > > > stored within SLRU segments today, we can extend pg_upgrade to
> > > > translate the segment files by scanning from relfrozenxid and
> > > > relminmxid and recording the corresponding values at the new
> > > > offsets in the target segments.
> > >
> > > +1
> > >
> > > (Hmm, if we're going to change all this stuff, I wonder if there would
> > > be any benefit to switching to 64 bit xids for the xid-based SLRUs
> > > while we're here...)
> >
> > Do you mean switching or reserving space for it on the block? The latter
> > I hope :-)
>
> I'd make the addressing work in a way that never requires wraparounds,
> but instead allows trimming at the beginning. That shouldn't result in
> any additional space, while allowing to fully switch to 64bit xids.

+1, will keep this in mind during implementation.

--
Shawn Debnath
Amazon Web Services (AWS)