Streaming Logical Decoding Events

From: Matt Dee <mattdee123(at)gmail(dot)com>
To: "pgsql-general(at)postgresql(dot)org" <pgsql-general(at)postgresql(dot)org>
Subject: Streaming Logical Decoding Events
Date: 2018-08-11 13:34:04
Message-ID: CACdswHNZMUyqROY9Ne69rmNh5B88707u+B2CRQxgrOa0yxfPHw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi there,

I am trying to build an application which reads messages off of a
replication slot and does some processing with them. I want to guarantee
that every event gets processed exactly once. Unfortunately, I've been
unable to find a guide to the correct way to read this data. I've been
experimenting and it seems to work, but I wanted to confirm a few things.
Apologies if this is the wrong place for this kind of question, or if
reading off a slot is described elsewhere (if so, I would appreciate being
pointed to the right place).

I'm trying to use terminology from
https://www.postgresql.org/docs/9.5/static/protocol-replication.html

My basic idea is:
* On startup, create a replication connection and do START CONNECTION with
LSN 0/0. This seems to start off from the last "committed" LSN (last LSN
sent back in a standby status update).

* Keep track of a "last committed" LSN, and periodically send back a
Standby Status Update with the "written", "flushed", and "applied" LSNn all
set to that value (I could not find documentation on the difference between
the three, but setting all of them equal seems to work).

* When I see an XLogData message, process the data and then set the "last
committed" LSN to the "starting point of the WAL data" LSN.

* When I see a Primary Keepalive Message, set "last committed" LSN to the
"current end WAL" field, assuming all prior XLogDatas have been processed.
If the "should reply" byte is set, send back a Standby Status Update
immediately

Is this the way that this feature was intended to be used?

In particular, I have two concerns with this approach:

1) On the XLogData, should I be using "starting point of WAL data" or
"current end of WAL"? In my testing, they seem to always be the same, so
I'm worried there's some case I haven't handled.

2) I'm not sure if committing the end WAL LSNs on the Keepalive messages is
safe. Is it possible to receive a Keepalive message, followed by an
XLogData message, such that the Keepalive's LSN is higher than the
XLogData's? This could cause problems, because after committing the
Keepalive's LSN, if my application needed to restart, when it comes back
up, we might miss the XLogData, since we committed a higher LSN.

Thanks, and I appreciate any pointers you can provide!
-Matt

Browse pgsql-general by date

  From Date Subject
Next Message Devrim Gündüz 2018-08-11 14:39:33 Re: Who and How is responsible for released installations packages and 3rd party packs? (e.g. on https://yum.postgresql.org/9.6/redhat/rhel-7.3-x86_64/)
Previous Message Pavel Stehule 2018-08-11 06:14:49 Re: User documentation vs Official Docs