Re: BUG #6200: standby bad memory allocations on SELECT

On Fri, Jan 27, 2012 at 1:31 PM, Bridget Frey <bridget(dot)frey(at)redfin(dot)com> wrote:
> Thanks for the info - that's very helpful.  We had also noted that the alloc
> seems to be -3 bytes.  We have run pg_check and it found no instances of
> corruption. We've also replayed queries that have failed, and have never
> been able to get the same query to fail twice.  In the case you
> investigated, was there permanent page corruption - e.g. you could run the
> same query over and over and get the same result?

Yes. I observed that the infomask bits on several tuples had somehow
been overwritten by nonsense. I am not sure whether there were other
kinds of corruption as well - I suspect probably so - but that's the
only one I saw with my own eyes, courtesy of pg_filedump.

> It really does seem like this is an issue either in Hot Standby or very
> closely related to that feature, where there is temporary corruption of a
> btree index that then disappears.  Our master is not experiencing any malloc
> issues, while the 3 slaves get about a dozen per day, despite similar
> workloads.  We haven't have a slave segfault since we set it up to produce a
> core dump, but we're expecting to have that within the next few days
> (assuming we'll continue to get a segfault every 3-4 days).  We're also
> planning to set up one slave that will panic when it gets a malloc issue, as
> you (and other posters on 6400) had suggested.
>
> Thanks again for the help, and we'll keep you posted as we learn more...

The case I investigated involved corruption on the master, and I think
it predated Hot Standby. However, the symptom is generic enough that
it seems quite possible that there's more than one way for it to
happen. :-(

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company