Re: [PROPOSAL] Shared Ispell dictionaries

Hello hackers,

On 25.02.2019 14:33, Arthur Zakirov wrote:
>> The current approach inherently involves double-buffering: you've got
>> the filesystem cache containing the data read from disk, and then the
>> DSM containing the converted form of the data.  Having something that
>> you could just mmap() would avoid that, plus it would become a lot
>> less critical to keep the mappings around.  You could probably just
>> have individual queries mmap() it for as long as they need it and then
>> tear out the mapping when they finish executing; keeping the mappings
>> across queries likely wouldn't be too important in this case.
>>
>> The downside is that you'd probably need to teach resowner.c about
>> mappings created via mmap() so that you don't leak mappings on an
>> abort, but that's probably not a crazy difficult problem.
>
> It seems to me Tom and Andres also vote for the mmap() approach. I think
> I need to look closely at the mmap().
>
> I've labeled the patch as 'v13'.

I've attached new version of the patch. Note that it is in WIP state for
now and there are unresolved issues, which is listed at the end of the
email.

The patch implements simple approach of using mmap(). Also I want to be
sure that I'm going in right direction. Feel free to send a feedback.

On every dispell_init() call Postgres checks is there a shared
dictionary file in the pg_shdict directory, if it is then calls mmap().
If there is no such file then it compiles the dictionary, write it to
the file and calls mmap().

dispell_lexize() works with already mmap'ed dictionary. So it doesn't
mmap() for each individual query as Robert proposed above. It's because
such approach reduces performance twice (I tested with ts_lexize() calls
by pgbench).

Tests
-----

Like in:
https://www.postgresql.org/message-id/20180124172039.GA11210%40zakirov.localdomain

i performed tests. There are now big differences in numbers except that
files are being created now in the pg_shdict directory:

czech_hunspell - 9.2 MB file
english_hunspell - 1.9 MB file
french_hunspell - 4.6 MB file

TODO
----

- Improve the documentation and comments.
- Eliminate shared dictionary files after DROP/ALTER calls. It necessary
to come up with some fancy file name. For now it is just OID of a
dictionary. So it is possible to add database OID, xmin or xmax into a
file name.
- We cant remove the file right away after DROP/ALTER. Is it good idea
to use autovacuum here?

--
Arthur Zakirov
Postgres Professional: http://www.postgrespro.com
Russian Postgres Company