| From: | Peter Geoghegan <pg(at)bowt(dot)ie> |
|---|---|
| To: | Jeff Davis <pgsql(at)j-davis(dot)com> |
| Cc: | Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>, David Rowley <dgrowleyml(at)gmail(dot)com>, Stephen Frost <sfrost(at)snowman(dot)net>, Andres Freund <andres(at)anarazel(dot)de>, Bruce Momjian <bruce(at)momjian(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, Justin Pryzby <pryzby(at)telsasoft(dot)com>, Melanie Plageman <melanieplageman(at)gmail(dot)com>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org> |
| Subject: | Re: Default setting for enable_hashagg_disk |
| Date: | 2020-07-26 00:31:49 |
| Message-ID: | CAH2-Wznidojad-zbObnFOzDA5RTCS0JLsqcZkDNu+ou1NGYQYQ@mail.gmail.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-docs pgsql-hackers |
On Sat, Jul 25, 2020 at 4:56 PM Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> I wrote a quick patch to use HyperLogLog to estimate the number of
> groups contained in a spill file. It seems to reduce the
> overpartitioning effect, and is a more principled approach than what I
> was doing before.
This pretty much fixes the issue that I observed with overparitioning.
At least in the sense that the number of partitions grows more
predictably -- even when the number of partitions planned is reduced
the change in the number of batches seems smooth-ish. It "looks nice".
> It does seem to hurt the runtime slightly when spilling to disk in some
> cases. I haven't narrowed down whether this is because we end up
> recursing multiple times, or if it's just more efficient to
> overpartition, or if the cost of doing the HLL itself is significant.
I'm glad that this better principled approach is possible. It's hard
to judge how much of a problem this really is, though. We'll need to
think about this aspect some more.
Thanks
--
Peter Geoghegan
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Peter Geoghegan | 2020-07-26 00:52:11 | Re: Default setting for enable_hashagg_disk |
| Previous Message | Peter Geoghegan | 2020-07-26 00:13:00 | Re: Default setting for enable_hashagg_disk |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Peter Geoghegan | 2020-07-26 00:52:11 | Re: Default setting for enable_hashagg_disk |
| Previous Message | Peter Geoghegan | 2020-07-26 00:13:00 | Re: Default setting for enable_hashagg_disk |