Re: logical changeset generation v4 - Heikki's thoughts about the patch state

Hi,

I decided to reply on the patches thread to be able to find this later.

On 2013-01-23 22:48:50 +0200, Heikki Linnakangas wrote:
> "logical changeset generation v4"
> This is a boatload of infrastructure for supporting logical replication, yet
> we have no code actually implementing logical replication that would go with
> this. The premise of logical replication over trigger-based was that it'd be
> faster, yet we cannot asses that without a working implementation. I don't
> think this can be committed in this state.

Its a fair point that this is a huge amount of code without a user in
itself in-core.
But the reason it got no user included is because several people
explicitly didn't want a user in-core for now but said the first part of
this would be to implement the changeset generation as a separate
piece. Didn't you actually prefer not to have any users of this in-core
yourself?

Also, while the apply side surely isn't benchmarkable without any being
submitted, the changeset generation can very well be benchmarked.

A very, very adhoc benchmark:
-c max_wal_senders=10
-c max_logical_slots=10 --disabled for anything but logical
-c wal_level=logical --hot_standby for anything but logical
-c checkpoint_segments=100
-c log_checkpoints=on
-c shared_buffers=512MB
-c autovacuum=on
-c log_min_messages=notice
-c log_line_prefix='[%p %t] '
-c wal_keep_segments=100
-c fsync=off
-c synchronous_commit=off

pgbench -p 5440 -h /tmp -n -M prepared -c 16 -j 16 -T 30

pgbench upstream:
tps: 22275.941409
space overhead: 0%
pgbench logical-submitted
tps: 16274.603046
space overhead: 2.1%
pgbench logical-HEAD (will submit updated version tomorrow or so):
tps: 20853.341551
space overhead: 2.3%
pgbench single plpgsql trigger (INSERT INTO log(data) VALUES(NEW::text))
tps: 14101.349535
space overhead: 369%

Note that in the single trigger case nobody consumed the queue while the
logical version streamed the changes out and stored them to disk.

Adding a default NOW() or similar to the tables immediately makes
logical decoding faster by a factor of about 3 in comparison to the
above trivial trigger.

The only reason the submitted version of logical decoding is
comparatively slow is that its xmin update policy is braindamaged,
working on that right now.

Greetings,

Andres

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services