Re: logical changeset generation v3

On Wed, Nov 21, 2012 at 4:30 PM, Andres Freund <andres(at)2ndquadrant(dot)com>wrote:

> On 2012-11-21 15:28:30 +0900, Michael Paquier wrote:
> > On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund <andres(at)2ndquadrant(dot)com
> >wrote:
> >
> > > On 2012-11-20 09:30:40 +0900, Michael Paquier wrote:
> > > > Btw, here are some extra comments based on my progress, hope it will
> be
> > > > useful for other people playing around with your patches.
> > > > 1) Necessary to install the contrib module test_decoding on server
> side
> > > or
> > > > the test case will not work.
> > > > 2) Obtention of the following logs on server:
> > > > LOG: forced to assume catalog changes for xid 1370 because it was
> > > running
> > > > to early
> > > > WARNING: ABORT 1370
> > > > Actually I saw that there are many warnings like this.
> > >
> > > Those aren't unexpected. Perhaps I should not make it a warning then...
> > >
> > A NOTICE would be more adapted, a WARNING means that something that may
> > endanger the system has happened, but as far as I understand from your
> > explanation this is not the case.
>
> I think it should go DEBUG2 or so once were a bit more confident about
> the code.
>
> > > A short explanation:
> > >
> > > We can only decode tuples we see in the WAL when we already have a
> > > timetravel catalog snapshot before that transaction started. To build
> > > such a snapshot we need to collect information about committed which
> > > changed the catalog. Unfortunately we can't diagnose whether a txn
> > > changed the catalog without a snapshot so we just assume all committed
> > > ones do - it just costs a bit of memory. Thats the background of the
> > > "forced to assume catalog changes for ..." message.
> > >
> > OK, so this snapshot only needs to include the XIDs of transactions that
> > have modified the catalogs. Do I get it right? This way you are able to
> > fetch the correct relation definition for replication decoding.
>
> Yes. We only carry those between (recenXmin, newestCatalogModifyingTxn),
> so its not all of them. Normal snapshots carry all in-progress
> transactionids instead of the committed ones, but that would have been
> far more in our case (only a minority of txn's touch the catalog) and it
> has problems with subtransaction tracking.
>
Hum. I might have missed something but what is the variable tracking the
newest XID that modified catalogs.
I can see of course recentXmin in snapmgr.c but nothing related to what you
describe.

>
> > Just thinking but... It looks to be a waste to store the transactions
> XIDs
> > of all the committed transactions, but on the other hand there is no way
> to
> > track the XIDs of transactions that modified a catalog in current core
> > code. So yes this approach is better as refining the transaction XID
> > tracking for snapshot reconstruction is something that could be improved
> > later. Those are only thoughts though...
>
> We actually only track xids of catalog modifying transactions once we
> hit the CONSISTENT state. Before the initial snapshot we can't detect
> that.
>
How to you track them? I think I need to go deeper in the code before
asking more...
--
Michael Paquier
http://michael.otacoo.com