-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
> I'm going to look at mirroring it onto the same server that runs
> ftp.postgresql.org ... archives is the worst site that we run, since its
> all a bunch of little flat files, so when it gets indexed by the various
> search engines, disk I/O goes through the roof ... we had googlebot index
> it once where we had to literally shut down the server for a few minutes
> while we waited for load to drop ...
On googlebot's page[1], they claim they never go more than once every few
seconds. Surely this should not be a problem as long as these are static
pages. They also have an email address on that page where you can request
that Google go a little gentler on your site.
Also, if the pages are static (or static plus simple cgis), have you considered
using boa? [2] I use it for a large site that has a lot of static pages
and it does great - it's a small, clean, minimal web server written in C.
A final option is an accelerator cache [3]. Not sure if PG is using one
yet, but it probably should be.
[1] http://www.google.com/bot.html
[2] http://www.boa.org/
[3] http://www.squid-cache.org/Doc/FAQ/FAQ-20.html#what-is-httpd-accelerator
- --
Greg Sabino Mullane greg(at)turnstep(dot)com
PGP Key: 0x14964AC8 200408290732
-----BEGIN PGP SIGNATURE-----
iD8DBQFBMcEGvJuQZxSWSsgRAhK9AKCLQ4CIW2JQQDg+BFI12DyhaFiFVgCg1DW5
VkM2ayTI9OK6M1kIscAwgxs=
=1uGd
-----END PGP SIGNATURE-----