Skip site navigation (1) Skip section navigation (2)

Re: Archives too slow

From: "Greg Sabino Mullane" <greg(at)turnstep(dot)com>
To: pgsql-www(at)postgresql(dot)org
Subject: Re: Archives too slow
Date: 2004-08-29 11:41:14
Message-ID: 26f8860ca0585c01dd53ced6d9e9a336@biglumber.com (view raw or flat)
Thread:
Lists: pgsql-www
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
 
 
> I'm going to look at mirroring it onto the same server that runs
> ftp.postgresql.org ... archives is the worst site that we run, since its
> all a bunch of little flat files, so when it gets indexed by the various
> search engines, disk I/O goes through the roof ... we had googlebot index
> it once where we had to literally shut down the server for a few minutes
> while we waited for load to drop ...
 
On googlebot's page[1], they claim they never go more than once every few
seconds. Surely this should not be a problem as long as these are static
pages. They also have an email address on that page where you can request
that Google go a little gentler on your site.
 
Also, if the pages are static (or static plus simple cgis), have you considered
using boa? [2] I use it for a large site that has a lot of static pages
and it does great - it's a small, clean, minimal web server written in C.
 
A final option is an accelerator cache [3]. Not sure if PG is using one
yet, but it probably should be.
 
[1] http://www.google.com/bot.html
 
[2] http://www.boa.org/
 
[3] http://www.squid-cache.org/Doc/FAQ/FAQ-20.html#what-is-httpd-accelerator
 
- --
Greg Sabino Mullane greg(at)turnstep(dot)com
PGP Key: 0x14964AC8 200408290732
-----BEGIN PGP SIGNATURE-----
 
iD8DBQFBMcEGvJuQZxSWSsgRAhK9AKCLQ4CIW2JQQDg+BFI12DyhaFiFVgCg1DW5
VkM2ayTI9OK6M1kIscAwgxs=
=1uGd
-----END PGP SIGNATURE-----



In response to

Responses

pgsql-www by date

Next:From: Justin CliftDate: 2004-08-29 12:41:18
Subject: Re: Techdocs site down (db error)
Previous:From: Marc G. FournierDate: 2004-08-29 05:18:29
Subject: Re: Techdocs site down (db error)

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group