Skip site navigation (1) Skip section navigation (2)

Re: Load Distributed Checkpoints, take 3

From: Heikki Linnakangas <heikki(at)enterprisedb(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Greg Smith <gsmith(at)gregsmith(dot)com>, Patches <pgsql-patches(at)postgresql(dot)org>
Subject: Re: Load Distributed Checkpoints, take 3
Date: 2007-06-26 17:35:32
Message-ID: 46814E64.6030800@enterprisedb.com (view raw or flat)
Thread:
Lists: pgsql-patches
Tom Lane wrote:
> Heikki Linnakangas <heikki(at)enterprisedb(dot)com> writes:
>> Tom Lane wrote:
>>> Who's "we"?  AFAICS, CVS HEAD will treat a large copy the same as any
>>> other large heapscan.
> 
>> Umm, I'm talking about populating a table with COPY *FROM*. That's not a 
>> heap scan at all.
> 
> No wonder we're failing to communicate.  I assumed you were talking
> about copy-to-file.  Copy-from-file is going to be creating WAL entries
> hence the no-checkpoint case doesn't apply anyway, no?

We are indeed having trouble to communicate :(.

No, I'm not talking about the new non-WAL-logged COPY optimization. COPY 
FROM *would* create WAL entries, and the next checkpoint would clean 
them. So far so good. But then you run VACUUM, as you often do after 
loading a table, to set all hint bits. That will *not* generate WAL, and 
next checkpoint is skipped.

To recap, the sequence is:

1. COPY FROM
2. checkpoint
3. VACUUM

Now you have buffer cache full of dirty buffers with usage_count=1, and 
no WAL activity since last checkpoint. They will not be flushed until:
a) WAL activity happens and next checkpoint comes
b) database is shut down, or manual CHECKPOINT is issued
c) clock hand advances and decrements the usage_counts

It's a corner case. Probably not a problem in practice; you usually run 
CREATE INDEX after loading a table, for example. But it exists.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com

In response to

Responses

pgsql-patches by date

Next:From: Tom LaneDate: 2007-06-26 17:57:56
Subject: Re: Load Distributed Checkpoints, take 3
Previous:From: Tom LaneDate: 2007-06-26 17:23:05
Subject: Re: Load Distributed Checkpoints, take 3

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group