Skip site navigation (1) Skip section navigation (2)

Re: archived WALL files question

From: "Vitaly Burshteyn" <vburshteyn(at)broadway(dot)com>
To: <scott(dot)lists(at)enterprisedb(dot)com>,<renato(dot)oliveira(at)grant(dot)co(dot)uk>
Cc: <pgsql-admin(at)postgresql(dot)org>
Subject: Re: archived WALL files question
Date: 2010-04-15 16:07:34
Message-ID: 0F69574F9901D4459C6B75C9FF64FBC403E6781D@mxfl01.hollywoodmedia.corp (view raw or flat)
Thread:
Lists: pgsql-admin
We manage the WAL files via skytools WALMGR

As far as log files we run a backup every 3 house and keep 12 housers worth on the server, everything else is sent to amazon S3 via s3sync

________________________________

From: pgsql-admin-owner(at)postgresql(dot)org <pgsql-admin-owner(at)postgresql(dot)org> 
To: Renato Oliveira <renato(dot)oliveira(at)grant(dot)co(dot)uk> 
Cc: pgsql-admin(at)postgresql(dot)org <pgsql-admin(at)postgresql(dot)org> 
Sent: Thu Apr 15 11:56:44 2010
Subject: Re: [ADMIN] archived WALL files question 



On Thu, Apr 15, 2010 at 11:31 AM, Renato Oliveira <renato(dot)oliveira(at)grant(dot)co(dot)uk> wrote:


	

	Dear all,

	 

	I was reading again the documentation... “The archive command should generally de designed to refuse to overwrite any pre-existing archive file.”

	This means it will keep writing logs to the folder specified forever, and without an intervention, the media will run out of space.

	 

	What do you guys do with regards to this situation, for example:

	How to you clean up the old archived logs?

	 

	For example:

	you archive your log files from your main Postgres server to a folder /mnt/pitr for example

	You set your standby to pick the logs from /mnt/pitr, then it archives each log as it comes.

	/mnt/pitr will fill up very quickly and run out of space if we don’t have a process to DELETE/ARCHIEVE older logs.

	 

	I guess the process which picks up the logs for the standby server, needs to take care of the logs, by deleting the older ones or by archiving them permanently?

Depends on what it is you're trying to accomplish:


*) PITR slave server constantly applying logs

   If all you want is a server to constantly apply the logs and you don't care about them afterwards, look into the '%r' macro in pg_standby.  It will automatically archive files for you -- Of course, your standby instance needs to have write access to the /mnt/pitr folder to delete from.

   If you are using the archive_command to copy files in the /mnt/pitr directory, and then doing a cron based copy to a backup server, have your cronjob delete files from the primary after it is confirmed that the logs got shipped safely to the backup.

*) Backup retention time

   If you're trying to keep logs around so that you can do a point in time recovery with old backups, you want to figure your retention times and determine your RTO (http://en.wikipedia.org/wiki/Recovery_time_objective).

     If you need to be able to recovery to any point in time for the past 1 week with a low RTO, then you want to keep that week's worth of logs uncompressed and available.  Anything beyond that, use a cron job to compress the logs (they usually compress pretty well based on your data).  

     Basically, you need to keep all the low-RTO required logs around so that you can quickly get at them.  If you don't have any low RTO requirements and you just want to keep a few weeks worth of data around, I would recommend that you add a few lines of code to the end of your backup job to compress (or you could delete if you don't want them) all the logs prior to the backup that you are taking.

Hope this helps

--Scott
 

	

	 

	How do you guys deal with this problem?

	 

	Thank you very much in advance

	 

	Best regards

	 

	Renato

	 

	 

	 
	Renato Oliveira
	Systems Administrator
	e-mail: renato(dot)oliveira(at)grant(dot)co(dot)uk
	 
	Tel: +44 (0)1763 260811
	Fax: +44 (0)1763 262410
	www.grant.co.uk <http://www.grant.co.uk/> 
	 
	Grant Instruments (Cambridge) Ltd 
	 
	Company registered in England, registration number 658133
	 
	Registered office address:
	29 Station Road, 
	Shepreth, 
	CAMBS SG8 6GB 
	UK
	 
	
	 
	
	 
	
	 

	 

	P Please consider the environment before printing this email

	CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies. 
	 
	
	VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).
	 
	
	OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our website <http://www.grant.co.uk/Support/openxml.html> 



____________________________________
The information contained in this transmission may contain privileged 
and confidential information.  It is intended only for the use of the 
person(s) named above. If you are not the intended recipient,  you are 
hereby notified that any review, dissemination, distribution or 
duplication of this communication is strictly prohibited. If you are 
not the intended recipient, please contact the sender by reply email 
and destroy all copies of the original message.

Responses

pgsql-admin by date

Next:From: Kevin GrittnerDate: 2010-04-15 17:27:31
Subject: Re: Query is stuck
Previous:From: Kevin GrittnerDate: 2010-04-15 16:01:49
Subject: Re: archived WALL files question

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group