Re: Minimal logical decoding on standbys

I am yet to work on Andres's latest detailed review comments, but I
thought before that, I should submit a patch for the below reported
issue because I was almost ready with the fix. Now I will start to
work on Andres's comments, for which I will reply separately.

On Fri, 1 Mar 2019 at 13:33, tushar <tushar(dot)ahuja(at)enterprisedb(dot)com> wrote:
>
> Hi,
>
> While testing this feature found that - if lots of insert happened on
> the master cluster then pg_recvlogical is not showing the DATA
> information on logical replication slot which created on SLAVE.
>
> Please refer this scenario -
>
> 1)
> Create a Master cluster with wal_level=logcal and create logical
> replication slot -
> SELECT * FROM pg_create_logical_replication_slot('master_slot',
> 'test_decoding');
>
> 2)
> Create a Standby cluster using pg_basebackup ( ./pg_basebackup -D
> slave/ -v -R) and create logical replication slot -
> SELECT * FROM pg_create_logical_replication_slot('standby_slot',
> 'test_decoding');
>
> 3)
> X terminal - start pg_recvlogical , provide port=5555 ( slave
> cluster) and specify slot=standby_slot
> ./pg_recvlogical -d postgres -p 5555 -s 1 -F 1 -v --slot=standby_slot
> --start -f -
>
> Y terminal - start pg_recvlogical , provide port=5432 ( master
> cluster) and specify slot=master_slot
> ./pg_recvlogical -d postgres -p 5432 -s 1 -F 1 -v --slot=master_slot
> --start -f -
>
> Z terminal - run pg_bench against Master cluster ( ./pg_bench -i -s 10
> postgres)
>
> Able to see DATA information on Y terminal but not on X.
>
> but same able to see by firing this below query on SLAVE cluster -
>
> SELECT * FROM pg_logical_slot_get_changes('standby_slot', NULL, NULL);
>
> Is it expected ?

Actually it shows up records after quite a long time. In general,
walsender on standby is sending each record after significant time (1
sec), and pg_recvlogical shows all the inserted records only after the
commit, so for huge inserts, it looks like it is hanging forever.

In XLogSendLogical(), GetFlushRecPtr() was used to get the flushed
point. On standby, GetFlushRecPtr() does not give a valid value, so it
was wrongly determined that the sent record is beyond flush point, as
a result of which, WalSndCaughtUp was set to true, causing
WalSndLoop() to sleep for some duration after every record. This is
why pg_recvlogical appears to be hanging forever in case of huge
number of rows inserted.

Fix : Use GetStandbyFlushRecPtr() if am_cascading_walsender.
Attached patch v8.

--
Thanks,
-Amit Khandekar
EnterpriseDB Corporation
The Postgres Database Company