|From:||Alexander Kuzmenkov <a(dot)kuzmenkov(at)postgrespro(dot)ru>|
|Subject:||index-only count(*) for indexes supporting bitmap scans|
|Views:||Raw Message | Whole Thread | Download mbox | Resend email|
I would like to propose a patch that speeds up the queries of the form
count(*) ... where ...', where the restriction clauses can be satisfied
indexes. At the moment, such queries use index-only scans followed by
aggregation. Index-only scans are only possible for indexes that are
returning indexed tuples, that is, support the 'amgettuple' access
are not applicable to indexes such as GIN and RUM. However, it is
improve count(*) performance for indexes that support bitmap scans. Having
performed a bitmap index scan or a combination of such, the bits in
be counted to obtain the final result. Of course, the bitmap pages that are
lossy or not visible to all existing transactions will still require heap
One kind of applications that can benefit from this change is the full-text
search with pagination. To show a search results page, the application
know the results that go to current page, and the total number of the
Getting one page is fast, when the desired sorting order can be provided
index. For example, results can be sorted by date with a separate btree
or by relevance with RUM index. However, getting the total number of
more difficult. With text search indexes, it requires a bitmap heap
can be rather slow due to obligatory heap access. A well-known hack for
using the approximate data from 'explain' results. The proposed change
the user to obtain the precise number of the results in an efficient and
The performance of this approach was tested on an archive of pgsql-hackers
mailing list. The detailed results for two sample queries can be found
attached file 'benchmark.txt'. The first test demonstrates full-text search
with RUM index, ordering the results by rank. For a query with low
getting the top results is much faster than counting them all with a
scan. With bitmap count execution plan, the results can be counted much
A similar test is done with a GIN index, where the results are
ordered by date using another btree index. Again, it shows a significant
for count(*) query for bitmap count scan compared to bitmap heap scan. These
results demonstrate that the bitmap count plan can indeed be useful for
text search scenarios.
Structurally, the patch consists of two major parts: a specialized executor
node and the generation of corresponding paths and plans. The optimizer
here is similar to that of the min/max aggregate optimization. The main
point is preprocess_count_aggs(); it is called by grouping_planner(). It
searches for count(*) expression in the target list, and, if possible,
a special path for it (struct BitmapCountPath). This path is then added to
UPPERREL_GROUP_AGG upperrel, and competes with other paths at the
The executor node (nodeBitmapCount.c) is similar to the bitmap heap scan
with the main difference being that it does not access heap for tuples
known to satisfy the restriction and to be visible to all transactions.
This patch has some important limitations:
* It only supports targetlist consisting of a single expression that can be
projected from count(*).
* count(expr) is not supported. We could support it for cases where the
"expr is not null" restriction can be satisfied with an index.
* The current implementation does not support parallel execution. It
implemented during the PostgreSQL 11 release cycle.
* For some indexes, the bitmap index scan will always require rechecking
the tuples. Bitmap count plans should not be used in such cases. For
check is not implemented.
I would be glad to hear your comments on this patch.
|Next Message||Masahiko Sawada||2017-04-11 17:36:18||Re: Quorum commit for multiple synchronous replication.|
|Previous Message||Pavan Deolasee||2017-04-11 17:20:51||Re: Patch: Write Amplification Reduction Method (WARM)|