Re: no mailing list hits in google

Hi,

On 2019-08-28 10:26:35 -0700, Andres Freund wrote:
> On August 28, 2019 9:22:44 AM PDT, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
> >Hackers,
> >[apologies if this is the incorrect list or is already discussed
> >material]
>
> Probably should be on the -www list. Redirecting. Please trim in future replies.
>
> >I've noticed that mailing list discussions in -hackers and other
> >mailing lists appear to not be indexed by google -- at all.
>
> I noticed that there's fewer and fewer hits too. Pretty annoying. I have an online archive I can search, but that's not something everyone should have to do.
>
> I think it's because robots.txt tells search engines to ignore the lists. Quite hard to understand how that's a good idea.
>
> https://www.postgresql.org/robots.txt
>
> User-agent: *
> Disallow: /admin/
> Disallow: /account/
> Disallow: /docs/devel/
> Disallow: /list/
> Disallow: /search/
> Disallow: /message-id/raw/
> Disallow: /message-id/flat/
>
> Sitemap: https://www.postgresql.org/sitemap.xml
>
> Without /list, there's no links to the individual messages. So there needs to be another external reference for a search engine to arrive at individual messages.
>
> Andres
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

For reasons that I do not understand, the previous mail had a broken
html part, making the above message invisible for people viewing the
html part.

Greetings,

Andres Freund