Re: Detecting File Damage & Inconsistencies

On Tue, Jul 6, 2021 at 4:21 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Fri, Jul 2, 2021 at 8:29 PM Simon Riggs <simon(dot)riggs(at)enterprisedb(dot)com> wrote:
> >
> > On Fri, Jul 2, 2021 at 5:34 AM Craig Ringer
> > <craig(dot)ringer(at)enterprisedb(dot)com> wrote:
> > >
> >
> > > If you don't think the sorts of use cases I presented are worth the trouble that's fair enough. I'm not against adding it on the commit record. It's just that with logical decoding moving toward a streaming model I suspect only having it at commit time may cause us some pain later.
> >
> > I think you have some good ideas about how to handle larger
> > transactions with streaming. As a separate patch it might be worth
> > keeping track of transaction size and logging something when a
> > transaction gets too large.
> >
>
> If we want this additional information for streaming mode in logical
> replication then can't we piggyback it on the very first record
> written for a transaction when this info is required? Currently, we do
> something similar for logging top_level_xid for subtransaction in
> XLogRecordAssemble().

It's possible, but I'm struggling to believe anybody would accept that
as an approach because it breaks simplicity, modularity and makes it
harder to search for this info in the WAL.

I was imagining that we'd keep track of amount of WAL written by a
transaction and when it reaches a certain size generate a "streaming
info" record as an early warning that we have a big transaction coming
down the pipe.

I'm feeling that a simple patch is expanding well beyond its original
scope and timeline. How can we do this simply?

--
Simon Riggs http://www.EnterpriseDB.com/