| From: | Craig Ringer <craig(at)2ndquadrant(dot)com> |
|---|---|
| To: | Andres Freund <andres(at)2ndquadrant(dot)com> |
| Cc: | Greg Stark <stark(at)mit(dot)edu>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org> |
| Subject: | Re: robots.txt on git.postgresql.org |
| Date: | 2013-07-10 08:25:13 |
| Message-ID: | 51DD1A69.1000001@2ndquadrant.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On 07/09/2013 11:30 PM, Andres Freund wrote:
> On 2013-07-09 16:24:42 +0100, Greg Stark wrote:
>> I note that git.postgresql.org's robot.txt refuses permission to crawl
>> the git repository:
>>
>> http://git.postgresql.org/robots.txt
>>
>> User-agent: *
>> Disallow: /
>>
>>
>> I'm curious what motivates this. It's certainly useful to be able to
>> search for commits.
>
> Gitweb is horribly slow. I don't think anybody with a bigger git repo
> using gitweb can afford to let all the crawlers go through it.
Wouldn't whacking a reverse proxy in front be a pretty reasonable
option? There's a disk space cost, but using Apache's mod_proxy or
similar would do quite nicely.
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Atri Sharma | 2013-07-10 08:32:24 | Re: Removing Inner Joins |
| Previous Message | Greg Jaskiewicz | 2013-07-10 08:03:20 | Listen/notify across clusters |