Re: bad wal on replica / incorrect resource manager data checksum in record / zfs

On Thu, Apr 2, 2020 at 2:10 PM Andres Freund <andres(at)anarazel(dot)de> wrote:

> Hi,
>
> On 2020-02-19 16:35:53 -0500, Alex Malek wrote:
> > We are having a reoccurring issue on 2 of our replicas where replication
> > stops due to this message:
> > "incorrect resource manager data checksum in record at ..."
>
> Could you show the *exact* log output please? Because this could
> temporarily occur without signalling anything bad, if e.g. the
> replication connection goes down.
>

Feb 23 00:02:02 wrds-pgdata10-2-w postgres[68329]: [12491-1] 5e4aac44.10ae9
(@) LOG: incorrect resource manager data checksum in record at
39002/57AC0338

When it occurred replication stopped. The only way to resume replication
was to stop server and remove bad WAL file.

>
>
> > Right before the issue started we did some upgrades and altered some
> > postgres configs and ZFS settings.
> > We have been slowly rolling back changes but so far the the issue
> continues.
> >
> > Some interesting data points while debugging:
> > We had lowered the ZFS recordsize from 128K to 32K and for that week the
> > issue started happening every other day.
> > Using xxd and diff we compared "good" and "bad" wal files and the
> > differences were not random bad bytes.
> >
> > The bad file either had a block of zeros that were not in the good file
> at
> > that position or other data. Occasionally the bad data has contained
> > legible strings not in the good file at that position. At least one of
> > those exact strings has existed elsewhere in the files.
> > However I am not sure if that is the case for all of them.
> >
> > This made me think that maybe there was an issue w/ wal file recycling
> and
> > ZFS under heavy load, so we tried lowering
> > min_wal_size in order to "discourage" wal file recycling but my
> > understanding is a low value discourages recycling but it will still
> > happen (unless setting wal_recycle in psql 12).
>
> This sounds a lot more like a broken filesystem than anythingon the PG
> level.
>

Probably. In my recent updated comment turning off ZFS compression on
master seems to have fixed the issue.
However I will note that the WAL file stored on the master was always fine
upon inspection.

>
>
> > When using replication slots, what circumstances would cause the master
> to
> > not save the WAL file?
>
> What do you mean by "save the WAL file"?
>

Typically, when using replication slots, when replication stops the master
will save the next needed WAL file.
However once or twice when this error occurred the master recycled/removed
the WAL file needed.
I suspect perhaps b/c the replica had started to read the WAL file it sent
some signal to the master that the WAL
file was already consumed. I am guessing, not knowing exactly what is
happening and w/ the caveat that this
situation was rare and not the norm. It is also possible caused by a
different error.

Thanks.
Alex