Re: Logical replication keepalive flood

On Sun, Sep 19, 2021 at 3:47 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> >
> > Yes, pg_recvlogical seems to be relying on receiving a keepalive for
> > its "--endpos" logic to work (and the 006 test is relying on '' record
> > output from pg_recvlogical in this case).
> > But is it correct to be relying on a keepalive for this?
> >
>
> I don't think this experiment/test indicates that pg_recvlogical's
> "--endpos" relies on keepalive. It would just print the records till
> --endpos and then exit. In the test under discussion, as per
> confirmation by Hou-San, the BEGIN record received has the same LSN as
> --endpos, so it would just output that and exit which is what is
> mentioned in pg_recvlogical docs as well (If there's a record with LSN
> exactly equal to lsn, the record will be output).
>

It seems to be relying on keepalive to get ONE specific record per
--endpos value, because once we apply the
"v1-0002-WIP-skip-the-keepalive-on-FIRST-loop-iteration.patch" patch,
then when pg_recvlogical is invoked for a second time with the same
--endos, it outputs the next record (BEGIN) too. So now for the same
--endpos value, we've had two different records output by
pg_recvlogical.
I have not seen this described in the documentation, so I think it
will need to be updated, should keepalives be reduced as per the
patch. The current documentation seems to be implying that one
particular record will be returned for a given --endpos (at least,
there is no mention of the possibility of different records being
output for the one --endpos, or that the first_lsn of a transaction is
always equal to end_lsn of the previous transaction).

--endpos=lsn

In --start mode, automatically stop replication and exit with
normal exit status 0 when receiving reaches the specified LSN. If
specified when not in --start mode, an error is raised.

If there's a record with LSN exactly equal to lsn, the record will be output.

The --endpos option is not aware of transaction boundaries and may
truncate output partway through a transaction. Any partially output
transaction will not be consumed and will be replayed again when the
slot is next read from. Individual messages are never truncated.

Regards,
Greg Nancarrow
Fujitsu Australia