robust archiving of WAL segments

I have a Postgres 8.4.X database that is subject to a fairly high
ingest rate. It is not uncommon for the system to generate 1-3 WAL
segments of 16M in size every minute. I endeavor to keep these WAL
segments around both for maintaining standby servers as well as for
disaster recovery.

I have over time ended up with a server architecture that looks like
this:

* master database has an archive_command that copies WAL segments
out of pg_xlog and puts them in another local directory that we'll
call wal_segment_staging_area

* there is a daemon that shovels files out of
wal_segment_staging_area using rsync and places them on a separate
file server in a directory we'll call wal_segment_archive

* there are standby servers in continuous recovery mode whose
restore_command sources WAL segments from the file server's
wal_segment_archive directory

The major considerations that governed this design are:

* the high rate of inflow generating a large number of segments quickly

* the desire not to have the WAL segments hosted on the master
database server long-term due to performance and fault-tolerance
reasons

* the fear of having an archive_command require network connectivity
to another server, the connectivity getting severed, and pg_xlog
getting filled and wedging the master server

* the problem that if you get substantially behind due to a
transient networking problem you want to be able to catch up as
quickly as possible, which means among other things that you don't
want to be opening up a separate remote connection to transfer
every segment in a loop, hence rsync

Over the past couple of years that this arrangement has been in
operation I have had on a handful of occasions occur what I believe to
be the following series of events:

* transfer operation of a WAL segment occurs, either from pg_xlog to
wal_segment_staging_area or from wal_segment_staging_area to
wal_segment_archive

* the process transferring the file believes that the operation has
succeeded and deletes its local copy of the file

* the server that received the file experiences a transient fault
that causes it to wedge and an administrator does a hard reset out
of desperation

* when the receiving server comes back up, the file that was created
during the archive operation is there, but it is zero-length,
presumably indicating that from the standpoint of an application
the write operation had succeeded, but from the standpoint of
caching the file's contents were not durably sync'd to disk

* when this has occurred to the file being placed in
wal_segment_staging_area, a subsequent power-on of the master
database has yielded repeated error messages in the log files
saying that the archive_command had failed (because, following the
Postgres docs' advice, the archive_command does a test to see if
the file already exists), and eventually Postgres says "I give up
trying to archive this WAL segment!" and deletes it from pg_xlog
without it being archived, which blew my mind the first time I saw
it happen because I thought the archive_command was supposed to be
retried indefinitely until success was reported

* when this has occurred to the file being placed in
wal_segment_archive, the effect is similar, although you don't get
any error messages in Postgres, just an archive that contains some
zero-length WAL segments

* in both cases, you end up with a WAL segment stream that is
corrupted, such that until you have made a new base backup if you
were to have to perform a point-in-time restore operation you
would only be able to restore up until the point of the event
because a subset of the WAL is lost forever

Thoughts and questions:

* it seems that robust archiving of WAL segments requires that your
script be as pedantic about transferring the files as Postgres is
about creating them; Postgres doesn't report a transaction as
having closed until the relevant WAL data has been durably written
to disk, and your archiving scripts cannot afford to delete the
source file until the destination file has likewise been durably
written

* is it a bug that Postgres gives up trying to archive a WAL segment
after several failed tries? or have I somehow misconfigured
something? maybe after the fashion of implementing a
restore_command for continuous recovery mode, i.e. implementing a
restore_command that does not return until a file has been
restored, I ought likewise implement an archive_command that does
not return until the file has been successfully archived?

* the Postgres docs chapter "Continuous Archiving and Point-in-Time
Recovery" provides a lot of good information about rigging up an
environment for dealing with all this stuff, but it does not to my
knowledge give any treatment to such matters as I describe herein,
and that may have yielded a somewhat precarious situation for a
lot of installations; there probably ought be a treatment of
ensuring that transmitted files have been written durably

* I'm not even sure what a good solution is for guaranteeing that
transmitted files have been durably persisted using common tools;
it doesn't seem that commonly available rsync implementations
support a "please call fsync" option, though some Googling yields
discussion of a patched version that someone created for such
purposes; maybe I could invoke the shell command "sync" as part of
the dance, but that doesn't seem that great either, since the
first transfer is happening on the master database and I don't
want to issue a sync request to all file systems as that will kill
database performance, and the second transfer is happening via
rsync and you wouldn't be able to call "sync" until "rsync" had
already deleted the source files, thus creating a race condition

* I can imagine what full solutions would look like, but not ones
that don't involve a fair amount of custom code to solve what
feels like it ought be a solved problem

How have you folks dealt with needing this level of robustness?

-- AWG