Re: no mailing list hits in google

Hi,

On August 28, 2019 9:22:44 AM PDT, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
>Hackers,
>[apologies if this is the incorrect list or is already discussed
>material]

Probably should be on the -www list. Redirecting. Please trim in future replies.

>I've noticed that mailing list discussions in -hackers and other
>mailing lists appear to not be indexed by google -- at all.

I noticed that there's fewer and fewer hits too. Pretty annoying. I have an online archive I can search, but that's not something everyone should have to do.

I think it's because robots.txt tells search engines to ignore the lists. Quite hard to understand how that's a good idea.

https://www.postgresql.org/robots.txt

User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /docs/devel/
Disallow: /list/
Disallow: /search/
Disallow: /message-id/raw/
Disallow: /message-id/flat/

Sitemap: https://www.postgresql.org/sitemap.xml

Without /list, there's no links to the individual messages. So there needs to be another external reference for a search engine to arrive at individual messages.

Andres
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.