Re: Opportunity for a Radical Changes in Database Software

On Oct 28, 2007, at 2:54 PM, Josh Berkus wrote:
> I'd actually be curious what incremental changes you could see
> making to
> PostgreSQL for better in-memory operation. Ideas?

It would be difficult to make PostgreSQL really competitive for in-
memory operation, primarily because a contrary assumption pervades
the entire design. You would need to rip out a lot of the guts of
it. I was not even intending to suggest that it would be a good idea
or trivial to adapt PostgreSQL to in-memory operation, but since I am
at least somewhat familiar with the research I thought I'd offer a
useful link that detailed the kinds of considerations involved. That
said, I have seriously considered the idea since I have a major
project that requires that kind of capability and there is some
utility in using parts of PostgreSQL if possible, particularly since
it was used to prototype it. In my specific case I also need to
shoehorn a new type of access method into it as well that there is no
conceptual support for, so it will probably be easier to build a
(mostly) new database engine altogether.

Personally, if I was designing a distributed in-memory database, I
would use a somewhat more conservative set of assumptions than
Stonebraker so that it would have a more general applicability. For
example, his assumption of extremely short CPU times for a
transaction (<1 millisecond) are not even valid for some types of
OLTP loads, never mind the numerous uses that are not strictly OLTP-
like but which nonetheless are built on relatively short
transactions; in the Stonebraker design this much latency would be a
pathology. Unfortunately, if you remove that assumption, the design
starts to unravel noticeably. Nonetheless, there are other viable
design paths that while not over-fitted to OLTP still could offer
large gains.

I think the market is right for a well-designed distributed, in-
memory database, but I think one would be starting with an
architecture inferior for the purpose that would be hard to get away
from if we made incremental changes to a solid disk-based engine. It
seems short-term expedient but long-term bad engineering -- think MySQL.

Cheers,

J. Andrew Rogers