Re: [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

On Monday, September 17, 2012 04:18:28 PM Heikki Linnakangas wrote:
> On 17.09.2012 17:08, Tom Lane wrote:
> > Heikki Linnakangas<hlinnakangas(at)vmware(dot)com> writes:
> >> On 17.09.2012 13:01, Andres Freund wrote:
> >>> It seems we would need one additional callback for both approaches
> >>> like: ->error(severity, format, ...)
> >>> For both to avoid having to draw in elog.c.
> >>
> >> Yeah. Another approach would be to return the error string from
> >> ReadRecord. The caller could then do whatever it pleases with it, like
> >> ereport() it to the log or PANIC. I think I'd like that better.
> >
> > I think it's basically insane to imagine that you can carve out a
> > non-trivial piece of the backend that doesn't contain any elog calls.
> > There's too much low-level infrastructure, such as palloc, that could
> > call it. Even if you managed to make it safe at the instant the feature
> > is committed, the odds it would stay safe over time are negligible.
>
> I wasn't thinking that we'd completely eliminate all elog() calls from
> ReadRecord and everything it calls, but only the "expected" ones that
> mean we've reached the end of valid WAL. The ones that use
> emode_for_corrupt_record(). Any unexpected errors like running out of
> file descriptors would still use ereport() like usual.
>
> That said, Andres' suggestion of making this facility completely
> independent of any backend functions, making it usable in external
> programs, doesn't actually seem that hard. ReadRecord() itself is fairly
> small, as are the subroutines that validate the records. XLogReadPage(),
> which goes out to fetch the right xlog page from archive or whatever, is
> way more complicated. But that would live in the callback, so it would
> be free to use all the normal backend facilities. However, it means that
> external programs would need to supply their own (hopefully much
> simpler) version of XLogReadPage(); I'm not sure how that goes with
> Andres' plans on using xlogreader.
XLogRead() from walsender.c is pretty easy to translate to backend-independent
code, so I don't think thats a problem. I don't see how the backend's version
is useful outside of the startup process anyway.

We could provide a default backend independent variant that hits files in
xlogreader.c, its not much code, to avoid others copying it multiple times.

I used a variant of that in the places that read from disk without any
problems. Obviously not in the places that read from network, but thats shelved
due to the different decoding approach atm anyway.

Regards,

Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services