Skip site navigation (1) Skip section navigation (2)

Re: problem with archive_command as suggested by documentation

From: "Albe Laurenz" <laurenz(dot)albe(at)wien(dot)gv(dot)at>
To: "Heikki Linnakangas *EXTERN*" <heikki(dot)linnakangas(at)enterprisedb(dot)com>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: problem with archive_command as suggested by documentation
Date: 2009-01-23 07:20:35
Message-ID: D960CB61B694CF459DCFB4B0128514C202FCC9E4@exadv11.host.magwien.gv.at (view raw or flat)
Thread:
Lists: pgsql-hackers
Heikki Linnakangas wrote:
> > The documentation states
> > 
> > "The archive command should generally be designed to refuse to overwrite any pre-existing archive file."
> > 
> > and suggests an archive_command like "test ! -f .../%f && cp %p .../%f".
> > 
> > We ran into (small) problems with an archive_command similar to this
> > as follows:
> > 
> > The server received a fast shutdown request while a WAL segment was being archived.
> > The archiver stopped and left behind a half-written archive file.
> 
> Hmm, if I'm reading the code correctly, a fast shutdown request 
> shouldn't kill an ongoing archive command.

Maybe it died because of a signal 1, I don't know.
But it left behind a half-written file.

> > Now when the server was restarted, the archiver tried to archive the same
> > WAL segment again and got an error because the destination file already
> > existed.
> > 
> > That means that WAL archiving is stuck until somebody manually removes
> > the partial archived file.
> 
> Yeah, that's a good point. Even if it turns out that the reason for your 
>   partial write wasn't the fast shutdown request, the archive_command 
> could be interrupted for some other reason and leave behind a partially 
> written file behind.
> 
> > I suggest that the documentation be changed so that it does not
> > recommend this setup. WAL segment names are unique anyway.
> 
> Well, the documentation states the reason to do that:
> 
> > This is an important safety feature to preserve the 
> integrity of your archive in case of administrator error 
> (such as sending the output of two different servers to the 
> same archive directory)
> 
> which seems like a reasonable concern too.

Of course, that's why I did that at first.

But isn't it true that the vast majority of people have only one
PostgreSQL cluster per machine, and it is highly unlikely that
somebody else creates a file with the same name as a WAL segment
in the archive directory?

> Perhaps it should suggest 
> something like:
> 
> test ! -f .../%f && cp %p .../%f.tmp && mv .../%f.tmp .../%f
> 
> ie. copy under a different filename first, and rename the file in place 
> after it's completely written, assuming that mv is atomic. It gets a bit 
> complicated, though.

That's a good idea (although it could lead to race conditions in the
extremely rare case that two clusters want to archive equally named
files at the same time).

I'll write a patch for that and send it as basis for a discussion.

Yours,
Laurenz Albe

In response to

Responses

pgsql-hackers by date

Next:From: Peter EisentrautDate: 2009-01-23 08:25:30
Subject: Re: autovacuum and reloptions
Previous:From: KaiGai KoheiDate: 2009-01-23 05:30:35
Subject: SE-PostgreSQL Updated Revision (r1460)

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group