Re: Report bytes and transactions actually sent downtream

On Tue, Jan 27, 2026 at 11:08 PM Andres Freund <andres(at)anarazel(dot)de> wrote:
>
> Hi,
>
> On 2025-12-11 10:29:42 +0530, Ashutosh Bapat wrote:
> > Please review.
>
> I'd simplify the patch to, initially, to just track the sent bytes. For one,
> that's by *far* the most useful statistic. But I also have some concerns
> about the other stats:
>

Ok. I have split the patch into two
0001 to track sent_bytes
0002 to track filtered_bytes and sent_txns

>
> - To me filteredBytes is a pretty bogus number - the size that the output
> plugin would have sent and ReorderBufferChangeSize() are only kinda
> related. It'll be a hard number to interpret, I think.
>
> It also seems to not account for filtering that happens based on the origin
> id.
>

total_bytes doesn't count the amount of changes that were never added
to the reorder buffer including those filtered based on the origin.
filtered_bytes is supposed to report the amount of changes out of
total_bytes that were filtered by the output plugin. It acts as an
indicator of filtering efficiency of a publication or whatever
filtering mechanism is employed by the output plugin.

>
> - I don't have fundamental opposition to tracking the number of sent
> transactions, but I think the implementation is at the wrong place.
>
> I think we ought to add explicit support for output plugins to filter
> transactions. Calling output plugins once for each change in a large
> transaction, which we already decided to not send out, makes no sense. It's
> far from free to do all the setup to decode a tuple, if the transaction is
> filtered, we shouldn't do that.
>
> It's also far far from free to restore changes from disk if we are going to
> throw away the whole transaction.

All the transactions that decided not to be sent out e.g. the
transactions committed before a confirmed_flush_lsn or the
transactions from filtered origins never reach the reorder buffer.
They don't have the problems you mention above. These transactions are
not subject of sent_txns at all.

>
> The current way requires each output plugin to maintain its own tracking
> about whether it decided to not output the transaction, which doesn't seem
> right to me.

The transactions which are sent downstream after applying transaction
level filtering by the output plugin are counted by sent_txns. Right
now these are the transactions whose all changes have been filtered
out by the output plugin. The output plugin can realize only at the
time of decoding the end of the transaction that it has not sent any
changes for that transaction and filter it out. It can not report
earlier than that. So I don't see any other way to implement the
counter.

If it would have been the case that the output plugin could know that
a transaction is going to get filtered completely when decoding one of
the changes, what you are suggesting would be helpful. But that's not
the case today.

>
> I'm also just not sure how useful it is, because most of the time we're
> going to filter on a per-change basis (e.g. only rels in a publication). But
> that's only sometimes going to affect the numbers of sent transactions.
>
>
> I'm not convinced as-is it's worth the breakage of all output plugins.

Hmm. I do see a usecase for both filtered_bytes and sent_txns as
indicators of filtering efficiency of output plugin. But not everybody
will see it worth breaking all output plugins.

I think we all agree that sent_bytes is useful, so I suggest that we
review and commit 0001. We can debate about 0002 after that.

--
Best Wishes,
Ashutosh Bapat