Re: no mailing list hits in google

Hi,

On 2019-08-29 13:12:00 +0200, Magnus Hagander wrote:
> On Wed, Aug 28, 2019 at 7:45 PM Andres Freund <andres(at)anarazel(dot)de> wrote:
>
> > Hi,
> >
> > On 2019-08-28 19:09:40 +0200, Magnus Hagander wrote:
> > > It blocks /list/ which has the subjects only.
> >
> > Yea. But there's no way to actually get to all the individual messages
> > without /list/? Sure, some will be linked to from somewhere else, but
> > without the content below /list/, most won't be reached?
> >
>
> That is indeed a good point. But it has been that way for many years, so
> something must've changed. We last modified this in 2013....

Hm. I guess it's possible that most pages were found due to the
next/prev links in individual messages, once one of them is linked from
somewhere externally. Any chance there's enough logs around to see
from where to where the indexers currently move?

> I wonder if we can inject these into Google using a sitemap. I think that
> should work -- will need some investigation on exactly how to do it, as
> sitemaps also have individual restrictions on the number of urls per file,
> and we do have quite a few messages.

Hm. You mean in addition to allowing /list/ or solely?

> > Why is that /list/ exclusion there in the first place?

> Because there are basically infinite number of pages in that space, due to
> the fact that you can pick an arbitrary point in time to view from.

You mean because of the per-day links, that aren't really per-day? I
think the number of links due to that would still be manageable traffic
wise? Or are they that expensive to compute? Perhaps we could make the
"jump to day" links smarter in some way? Perhaps by not including
content for the following days in the per-day pages?

Greetings,

Andres Freund