Re: Scalability in postgres

On Fri, 5 Jun 2009, Mark Mielke wrote:

> I disagree that profiling trumps theory every time.

That's an interesting theory. Unfortunately, profiling shows it doesn't
work that way.

Let's see if I can summarize the state of things a bit better here:

1) PostgreSQL stops working as efficiently with >1000 active connections

2) Profiling suggests the first barrier that needs to be resolved to fix
that is how the snapshots needed to support MVCC are derived

3) There are multiple patches around that aim to improve that specific
situation, but only being tested aggressively by one contributor so far
(that I'm aware of)

4) Those patches might cause a regression for other workloads, and the
section of code involved was very hard to get working well initially.
Before any change here will be accepted there needs to be a lot of data
proving it both does what expected and doesn't introduce a regression.

5) Few people are motivated to get their hands dirty doing the boring
benchmarking work to resolve this specific problem because "use a
connection pool" is a quite good workaround

6) Many other database vendors admit this problem so is hard to solve that
they also just suggest using a connection pool

If anyone wants to work on improving things here, (4) is the sticking
point that could use more hands. Not much theory involved, but there is a
whole lot of profiling.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD