Skip site navigation (1) Skip section navigation (2)

Re: Block-level CRC checks

From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Greg Stark <greg(dot)stark(at)enterprisedb(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Martijn van Oosterhout <kleptog(at)svana(dot)org>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Block-level CRC checks
Date: 2008-11-10 15:28:32
Message-ID: 20081110152832.GF6053@alvh.no-ip.org (view raw or flat)
Thread:
Lists: pgsql-hackers
Greg Stark wrote:
> I think double buffering solves the torn page problem but not the lack  
> of wal logging. Alvarro solved the wal logging  by deferring the wal  
> logs. But I'm not sure how confident we are that it's logging enough.
>

Right now, it's WAL-logging HeapTupleHeader hint bits (infomask and
infomask2), and ItemId (line pointer) flags.   Page pd_flags are skipped
in the CRC checksum -- this is easy to do because they are in a constant
offset in the page and I'm just skipping those bytes in CRC_COMP().

So what I'm missing is:
- btree hint bits
- bgwriter calls XLogInsert during shutdown, to WAL-log the hint bits
of unwritten pages.  This causes a PANIC to trigger about concurrent WAL
activity during checkpoint.  (The easy solution to this problem is just
to remove the check; another idea is to flush the buffers before
grabbing the final address to watch for at shutdown.)

> I'm beginning to think just excluding the hint bits would be simpler and 
> safer. If we're double buffering then it might be possible to do that 
> pretty cheaply. Copy the whole buffer with memcpy then loop through the 
> line pointers unsetting the hint bits. Then do the crc. Though that would 
> prevent us from doing "zero-copy" crc by doing it in the copy.

This can probably be made to work, and it solves the problem that
bgwriter calls XLogInsert during shutdown.  I would create new routines
to clear hint bits in all involved modules (heap_resethintbits, btree_%,
item_%, page_%), and call them on a copy of the page.

The downside to this idea is that we need to create a copy of the page
and call those routines when we read the page in, too.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

In response to

Responses

pgsql-hackers by date

Next:From: Alvaro HerreraDate: 2008-11-10 15:33:19
Subject: Re: per-database locale: createdb switches
Previous:From: Zdenek KotalaDate: 2008-11-10 14:56:07
Subject: pg_upgrade project status

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group