Re: Undo logs

On Fri, Aug 31, 2018 at 10:24 PM Dilip Kumar
<dilip(dot)kumar(at)enterprisedb(dot)com> wrote:
> On Fri, Aug 31, 2018 at 3:08 PM, Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> > As Thomas has already mentioned upthread that we are working on an
> > undo-log based storage and he has posted the patch sets for the lowest
> > layer called undo-log-storage.
> >
> > This is the next layer which sits on top of the undo log storage,
> > which will provide an interface for prepare, insert, or fetch the undo
> > records. This layer will use undo-log-storage to reserve the space for
> > the undo records and buffer management routine to write and read the
> > undo records.

I have also pushed a new WIP version of the lower level undo log
storage layer patch set to a public branch[1]. I'll leave the earlier
branch[2] there because the record-level patch posted by Dilip depends
on it for now.

The changes are mostly internal: it doesn't use DSM segments any more.
Originally I wanted to use DSM because I didn't want arbitrary limits,
but in fact DSM slots can run out in unpredictable ways, and unlike
parallel query the undo log subsystem doesn't have a plan B for when
it can't get the space it needs due to concurrent queries. Instead,
this version uses a pool of size 4 * max_connections, fixed at startup
in regular shared memory. This creates an arbitrary limit on
transaction size, but it's a large at 1TB per slot, can be increased,
doesn't disappear unpredictably, is easy to monitor
(pg_stat_undo_logs), and is probably a useful brake on a system in
trouble.

More soon.

[1] https://github.com/EnterpriseDB/zheap/tree/undo-log-storage-v2
[2] https://github.com/EnterpriseDB/zheap/tree/undo-log-storage

--
Thomas Munro
http://www.enterprisedb.com