Load distributed checkpoint

From: ITAGAKI Takahiro <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Load distributed checkpoint
Date: 2006-12-07 06:05:52
Message-ID: 20061207144843.6269.ITAGAKI.TAKAHIRO@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers pgsql-patches

This is a proposal for load distributed checkpoint.
(It is presented on postgresql anniversary summit in last summer.)

We offen encounters performance gap during checkpoint. The reason is write
bursts. Storage devices are too overworked in checkpoint, so they can not
supply usual transaction processing.

Checkpoint consists of the following four steps, and the major performance
problem is 2nd step. All dirty buffers are written without interval in it.

1. Query information (REDO pointer, next XID etc.)
2. Write dirty pages in buffer pool
3. Flush all modified files
4. Update control file

I suggested to write pages with sleeping in 2nd step, using normal activity
of the background writer. It is something like cost-based vacuum delay.
Background writer has two pointers, 'ALL' and 'LRU', indicating where to
write out in buffer pool. We can wait for the ALL clock-hand going around
to guarantee all pages to be written.

Here is pseudo-code for the proposed method. The internal loop is just the
same as bgwriter's activity.

PrepareCheckPoint(); -- do step 1
Reset num_of_scanned_pages by ALL activity;
do {
BgBufferSync(); -- do a part of step 2
sleep(bgwriter_delay);
} while (num_of_scanned_pages < shared_buffers);
CreateCheckPoint(); -- do step 3 and 4

We may accelerate background writer to reduce works at checkpoint instead of
the method, but it introduces another performance problem; Extra pressure
is always put on the storage devices to keep the number of dirty pages low.

I'm working about adjusting the progress of checkpoint to checkpoint timeout
and wal segments limitation automatically to avoid overlap of two checkpoints.
I'll post a patch sometime soon.

Comments and suggestions welcome.

Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jim C. Nasby 2006-12-07 06:32:56 Re: old synchronized scan patch
Previous Message Tom Lane 2006-12-07 05:46:34 Re: old synchronized scan patch

Browse pgsql-patches by date

  From Date Subject
Next Message Devrim GUNDUZ 2006-12-07 06:16:03 Re: [DOCS] 8.2.0 pdf
Previous Message Jim C. Nasby 2006-12-07 06:01:33 Re: 8.2rc1 (much) slower than 8.2dev?