Re: Inadequate hosting for www.postgresql.org

Its interesting to see how webperf.org sees different postgresql.org
pages. For example,

http://www.webperf.org/breakdown.html?URL=www.alexey.beta.postgresql.org
Connect DNS FirstByte End Size(Bytes) URL Status
0.033 0.074 3.226 4.225 17654 http://www.alexey.beta.postgresql.org 200
3.258 3.258 3.854 3.933 43 http://www.alexey.beta.postgresql.org/layout/images/0.gif 200
3.379 3.379 3.850 4.356 8190 http://www.alexey.beta.postgresql.org/layout/images/logo.png 200
3.519 3.520 3.930 4.070 2101 http://www.alexey.beta.postgresql.org/layout/css/new.css 200
3.710 3.729 4.067 4.363 11470 http://ads.postgresql.org/display_image.php?lvl=sponsor 200
3.935 3.936 8.678 8.801 2644 http://www.alexey.beta.postgresql.org/news.rss 200
4.072 4.072 7.736 7.739 864 http://www.alexey.beta.postgresql.org/events.rss 200
4.228 4.228 4.496 4.499 207 http://www.alexey.beta.postgresql.org/layout/images/mark_menu.png 200

It shows, that possible problem could be news.rss, events.rss.
www.postgresql.org behaves much better
http://www.webperf.org/breakdown.html?URL=http%3A%2F%2Fwww.postgresql.org
Connect DNS FirstByte End Size(Bytes) URL Status
0.033 0.074 0.403 1.118 21429 http://www.postgresql.org 200
0.417 0.417 1.082 1.099 929 http://www.postgresql.org/postgresql.css 200
0.581 0.581 1.220 1.354 4447 http://www.postgresql.org/images/banner-left.jpg 200
0.700 0.700 1.211 1.336 2929 http://www.postgresql.org/images/banner-right.jpg 200
0.823 0.841 1.259 1.522 14397 http://ads.postgresql.org/display_image.php?lvl=sponsor 200
0.955 0.955 1.252 1.498 14397 http://ads.postgresql.org/display_image.php?lvl=free 200

But, I again want to stress that current architecture of postgresql.org sites
are very-very bad and you will never get nice performance and scalability
especially for dynamic sites without:

1. Redirecting different traffic to different servers.

a)images and other binary things should be served by small and fast
server like thttpd or lighttpd. Currently, images are served by
apache+php server which is wasting of resources ! These servers
could easily serve several thousands req/sec, so one images.postgresql.org
could serve all such kind of traffic.

b)requests for dynamic pages should be forwarded to backend servers
with php compiled and keeping persistent connection to db.
These backends are fully hidden and accessable only by localhost,
so they will not stay busy while serving could-be-slow client
(slow connectivity), That means resources (memory) will be not wasted
and there is no demand to have many servers online, because interaction
with clients will be server only by frontends, see below

c)requests for static pages are served by plain httpd server like apache -
frontend. These servers should have cacheing capability to cache
answers from dynamic servers, so many-many requests for dynamic pages
will not actualy forwarded to heavy backends ! Our pages are not actually
very dynamic.

This is a classical 3 servers setup and it described in many books and
success stories.

Oleg
On Tue, 2 Nov 2004, Marc G. Fournier wrote:

> On Tue, 2 Nov 2004, Marc G. Fournier wrote:
>
>> On Tue, 2 Nov 2004, Alexey Borzov wrote:
>>
>>> Hi,
>>>
>>> Marc G. Fournier wrote:
>>>> You still haven't answered any of the questions that I asked in a private
>>>> email ... Is this a time of day issue (from your two samples, it looks
>>>> like it is)? If you set it up to run hourly, what do the #s look like
>>>> for each run throughout the day? What does the loadavg look like on the
>>>> server when you are running the script? Or is the database being slow?
>>>
>>> I tried running mirror later today:
>>>
>>> Nov 02 12:42:47 mirror [info] Mirroring finished. 423 page(s) saved, 2354
>>> second(s) spent
>>>
>>> I also am running it now, getting the same ~5 sec per page response time
>>> and uptime command states:
>>>
>>> 7:49PM up 10 days, 4:36, 4 users, load averages: 4.00, 3.22, 3.04
>>>
>>> The database itself does not look slow, nothing like these 5 seconds to
>>> connect / send the standard query.
>>>
>>>> Considering that Dave states above that the current script takes minutes
>>>> to generate >7000 pages, what are you doing differently that makes it so
>>>> much slower? And, why exactly are we changing from the current method if
>>>> the new method is going to require a dedicated server to run it?
>>>
>>> Yes, that's one way to ask these questions. The other way is: why are we
>>> having response times of ~5 seconds for not-too-complex pages?
>>
>> Your guess is as good as mine ... using my browser from here, I'm not seen
>> 5sec delays, and I'm two countries away from it ... using lynx on the VM
>> itself, goign through the web site, I'm seeing instantaneous responses ...
>> I'm definitely not seeing 5sec delays ...
>>
>> Can you extend your script so that it outputs URL spider'd and time taken
>> in msec for it? The pages that I'm testing definitely aren't giving me
>> 5sec lag times from the server itself ... so if you can give me a 'slow
>> URL' that I can look at, that would help ...
>
> 'k, I just looked at your mirror.php script .. and realized that its not the
> main web site thats the problem, its your new site ... why is your site so
> much slower to load up with lynx then the site that is there now?
> http://www.postgresql.org comes up instantly,
> http://www.alexey.beta.postgresql.org/index.html.en takes forever to load ...
> why?
>
> I can actually get coffee waiting for your site to load in my browser here,
> where there is almost zero delay loading the current site ...
>
> ----
> Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings
>

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83