Connection pooling for a mixture of lightweight and heavyweight jobs?

From: Craig James <craig_james(at)emolecules(dot)com>
To: pgsql-admin(at)postgresql(dot)org
Subject: Connection pooling for a mixture of lightweight and heavyweight jobs?
Date: 2010-07-28 22:13:57
Message-ID: 4C50ABA5.5010408@emolecules.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

I have a question that may be related to connection pooling.

We create a bunch of high-performance lightweight Postgres clients that serve up images (via mod_perl and Apache::DBI). We have roughly ten web sites, with ten mod_perl instances each, so we always have around 100 Postgres backends sitting around all the time waiting. When a lightweight request comes in, it's a single query on an primary key with no joins, so it's very fast.

We also have a very heavyweight process (our primary search technology) that can take many seconds, even minutes, to do a search and generate a web page.

The lightweight backends are mostly idle, but when a heavyweight search finishes, it causes a burst on the lightweight backends, which must be very fast. (They provide all of the images in the results page.)

This mixture seems to make it hard to configure Postgres with the right amount of memory and such. The primary query needs some elbow room to do its work, but the lightweight queries all get the same resources.

I figured that having these lightweight Postgres backends sitting around was harmless -- they allocate shared memory and other resources, but they never use them, so what's the harm? But recent discussions about connection pooling seem to suggest otherwise, that merely having 100 backends sitting around might be a problem.

Craig

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Bryan Payne 2010-07-29 14:00:00 Autovacuum missing tables
Previous Message Alex Hunsaker 2010-07-28 21:53:43 Re: postgresql-server-8.4.4-1PGDG.el5 - ERROR: could not write block 503414 of temporary file: No space le,ft on device