Re: pg_logical_emit_message() misses a XLogFlush()

On Mon, Sep 11, 2023 at 12:54:11PM +0900, Fujii Masao wrote:
> However, in the current patch, if "transactional" is set to false and
> "flush" is true, the function flushes the WAL immediately without
> considering synchronous_commit. Is this the intended behavior?
> I'm not sure how the function should work in this case, though.

Yes, that's the intended behavior. This just offers more options to
the toolkit of this function to give more control to applications when
emitting a message. In this case, like the current non-transactional
case, we make the record immediately available to logical decoders but
also make sure that it is flushed to disk. If one wants to force the
record's presence to a remote instance, then using the transactional
mode would be sufficient.

Perhaps you have a point here, though, that we had better make
entirely independent the flush and transactional parts, and still
call XLogFlush() even in transactional mode. One would make sure that
the record is on disk before waiting for the commit to do so, but
that's also awkward for applications because they would not know the
end LSN of the emitted message until the internal transaction commits
the allocated XID, which would be a few records after the result
coming out of pg_logical_emit_message().

The documentation does not worry about any of that even now in the
case of the non-transactional case, and it does not mention that one
may need to monitor pg_stat_replication or similar to make sure that
the LSN of the message exists on the remote with an application-level
check, either. How about adding an extra paragraph to the
documentation, then? I could think of something like that, but the
current docs also outline this a bit by telling that the message is
*not* part of a transaction, which kind of implies, at least to me,
that synchonous_commit is moot in this case:
"When transactional is false, note that the backend ignores
synchronous_commit as the record is not part of a transaction so there
is no commit to wait for. Ensuring that the record of a message
emitted exists on standbys requires additional monitoring."

> Though I don't understand the purpose of this option fully yet,
> is flushing the WAL sufficient? Are there scenarios where the function
> should ensure that the WAL is not only flushed but also replicated
> to the standby?

The flush makes sure that the record is durable, but we only care
about transaction commits in a synchronous setup, so that's
independent, in my opinion. If you look closely, we do some fancy
stuff in finish_sync_worker(), for example, where a transaction commit
is enforced to make sure that the internal flush is sensitive to the
synchronous commit requirements, but that's just something we expect
to happen in a sync worker.
--
Michael