Re: Pluggable Indexes (was Re: rmgr hooks (v2))

Hi all,

I hope to raise some valid concerns with the following two stories, a "true
story" first then a little fiction that I hope has a lot to do with current
reality.

Le jeudi 22 janvier 2009, Heikki Linnakangas a écrit :
> It's also impossible to do many other things without modifying the
> source code. Bitmap indexam had to do it, my clustered indexes had to do
> it, GIN had to do it.

So we're "only" talking about new index kinds which fit current indexam API,
right?

> Sure you can. Just Do It, if that's what you want. If you're willing to
> write a custom indexam, and run it in production, compiling PostgreSQL
> from source and patching it isn't such a stretch.

It's all about comfort and product maturity, isn't it?

I had performance concerns for prefix matching, ala telecom matches, i.e. the
prefix is in the table, not in the literal. And our IRC PostgreSQL guru told
me the best way to solve it would be implementing a specific datatype with
specific indexing facility. Easy enough? Sure, he said, just write an
external module and provide a GiST OPERATOR CLASS.
I did just this, wrote a single C file (less than 2000 lines) and I now run my
datatype and its GiST index in production. It has already served something
like 15 millions lookups and counting. Just works.
http://www.postgresql.org/docs/8.3/static/xindex.html
http://wiki.postgresql.org/images/3/30/Prato_2008_prefix.pdf
http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/prefix/prefix/

I assure you that should I have needed to patch PostgreSQL, I'd be running
plpgsql procedural code instead and would be fighting against this real time
costing evaluation trigger with such bad perfs.

True story. :)

> Don't get me wrong, I'm certainly not against pluggable indexes in
> principle. I just don't believe this patch brings us closer to that goal
> in any significant way.

If I understand the matter at all, it brings us closer only when the new index
type can be done without changing current indexam API. Which covers BTree,
Hash, GiST and GIN, so could probably cover some more.

If I were to start developping a new external module index kind, I'd really
like to avoid this situation:
- so for my new index idea, I'm only to write some custom C code?
- yes, an simple external module, following indexam API
- great, will I be able to WAL log it from this external module?
- of course, it's PostgreSQL we're talking about.
- what about recovering my custom index?
- oh. now you have to patch core code and run custom PG version
- huh?
- yes, core team finds the situation comfortable enough as is.
- ...

> Nothing stops you from starting right now, without this plugin. This is
> open source.

We're not talking about how great it is to be able to experiment new ideas by
forking core code, we're talking about making it easy and comfy to run user
code in production environments and being able to still apply minor upgrades
strait from the distribution.

Or maybe I'm misunderstanding it all.

Regards,
--
dim