Re: Time based lag tracking for logical replication

On Wed, May 3, 2017 at 6:28 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On 23 April 2017 at 01:10, Petr Jelinek <petr(dot)jelinek(at)2ndquadrant(dot)com> wrote:
>> Hi,
>>
>> The time based lag tracking commit [1] added interface for logging
>> progress of replication so that we can report lag as time interval
>> instead of just bytes. But the patch didn't contain patch for the
>> builtin logical replication.
>>
>> So I wrote something that implements this. I didn't like all that much
>> the API layering in terms of exporting the walsender's LagTrackerWrite()
>> for use by plugin directly. Normally output plugin does not have to care
>> if it's running under walsender or not, it uses abstracted write
>> interface for that which can be implemented in various ways (that's how
>> we implement SQL interface to logical decoding after all). So I decided
>> to add another function to the logical decoding write api called
>> update_progress and call that one from the output plugin. The walsender
>> then implements that new API to call the LagTrackerWrite() while the SQL
>> interface just does not implement it at all. This seems like cleaner way
>> of doing it.
>>
>> Thoughts?
>
> Agree cleaner.

+1

> I don't see any pacing or comments about it, nor handling of
> intermediate messages while we process a large transaction.
>
> I'll look at adding some pacing code in WalSndUpdateProgress()

By the way, I have a small improvement to the interpolation to
propose. Right now, after a period of idleness it can report a silly
large number based on an ancient time, but you won't usually see it
because it's quickly replaced by a sensible number. I think this
thinko will affect logical rep with Petr's patch more. I had been
meaning to post the improvement but got sidetracked by that recovery
test failure problem. I'll post that in the next few days.

--
Thomas Munro
http://www.enterprisedb.com