Re: logical changeset generation v3

On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund <andres(at)2ndquadrant(dot)com>wrote:

> Hi Michael,
>
>
> On 2012-11-19 16:28:55 +0900, Michael Paquier wrote:
> > I have been able to fetch your code (thanks Andrea!) and some it. For the
> > time being I am spending some time reading the code and understanding the
> > whole set of features you are trying to implement inside core, even if I
> > got some background from what you presented at PGCon and from the hackers
> > ML.
>
> Cool.
>
> > Btw, as a first approach, I tried to run the logical log receiver plugged
> > on a postgres server, and I am not able to make it work.
>
> > Well, I am using settings similar to yours.
> > # Run master
> > rm -r ~/bin/pgsql/master/
> > initdb -D ~/bin/pgsql/master/
> > echo "local replication $USER trust" >> ~/bin/pgsql/master/pg_hba.conf
> > postgres -D ~/bin/pgsql/master \
> > -c wal_level=logical \
> > -c max_wal_senders=10 \
> > -c max_logical_slots=10 \
> > -c wal_keep_segments=100 \
> > -c log_line_prefix="[%p %x] "
> > # Logical log receiver
> > pg_receivellog -f $HOME/output.txt -d postgres -v
> >
> > After launching some SQLs, the logical receiver is stuck just after
> sending
> > INIT_LOGICAL_REPLICATION, please see bt of process waiting:
>
> Its waiting till it sees initial an initial xl_running_xacts record. The
> easiest way to do that is manually issue a checkpoint. Sorry, should
> have included that in the description.
> Otherwise you can wait till the next routine checkpoint comes arround...
>
> I plan to cause more xl_running_xacts record to be logged in the
> future. I think the timing of those currently is non-optimal, you have
> the same problem as here in normal streaming replication as well :(
>
> > I am just looking at this patch and will provide some comments.
> > By the way, you forgot the installation part of pg_receivellog, please
> see
> > patch attached.
>
> That actually was somewhat intended, I thought people wouldn't like the
> name and I didn't want a binary that's going to be replaced anyway lying
> around ;)
>
OK no problem. For sure this is going to happen, I was wondering myself if
it could be possible to merge pg_receivexlog and pg_receivellog into a
single utility with multiple modes :)

Btw, here are some extra comments based on my progress, hope it will be
useful for other people playing around with your patches.
1) Necessary to install the contrib module test_decoding on server side or
the test case will not work.
2) Obtention of the following logs on server:
LOG: forced to assume catalog changes for xid 1370 because it was running
to early
WARNING: ABORT 1370
Actually I saw that there are many warnings like this.
3) Assertion failure while running pgbench, I was just curious to see how
it reacted when logical replication was put under a little bit of load.
TRAP: FailedAssertion("!(((xid) >= ((TransactionId) 3)) &&
((snapstate->xmin_running) >= ((TransactionId) 3)))", File: "snapbuild.c",
Line: 877)
=> pgbench -i postgres; pgbench -T 500 -c 8 postgres
4) Mentionned by Andres above, but logical replication begins only there is
a xl_running_xacts record. I just enforced a checkpoint manually.

With all those things done, I have been able to set up the system, for
example those queries:
postgres=# create table ac (a int);
CREATE TABLE
postgres=# insert into ac values (1);
INSERT 0 1
created the expected output:
BEGIN 32135
COMMIT 32135
BEGIN 32136
table "ac": INSERT: a[int4]:1
COMMIT 32136

Now it is time to dig into the code...
--
Michael Paquier
http://michael.otacoo.com