| From: | Jeff Davis <pgsql(at)j-davis(dot)com> |
|---|---|
| To: | Peter Geoghegan <pg(at)bowt(dot)ie> |
| Cc: | Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>, David Rowley <dgrowleyml(at)gmail(dot)com>, Stephen Frost <sfrost(at)snowman(dot)net>, Andres Freund <andres(at)anarazel(dot)de>, Bruce Momjian <bruce(at)momjian(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, Justin Pryzby <pryzby(at)telsasoft(dot)com>, Melanie Plageman <melanieplageman(at)gmail(dot)com>, "pgsql-hackers(at)postgresql(dot)org" <pgsql-hackers(at)postgresql(dot)org> |
| Subject: | Re: Default setting for enable_hashagg_disk |
| Date: | 2020-07-25 20:10:36 |
| Message-ID: | d5af7930265b8c4bbda5381364d7e21955597538.camel@j-davis.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-docs pgsql-hackers |
On Sat, 2020-07-25 at 11:05 -0700, Peter Geoghegan wrote:
> What worries me a bit is the sharp discontinuities when spilling with
> significantly less work_mem than the "optimal" amount. For example,
> with Tomas' TPC-H query (against my smaller TPC-H dataset), I find
> that setting work_mem to 6MB looks like this:
...
> Planned Partitions: 128 Peak Memory Usage: 6161kB Disk
> Usage: 2478080kB HashAgg Batches: 128
...
> Planned Partitions: 128 Peak Memory Usage: 5393kB Disk
> Usage: 2482152kB HashAgg Batches: 11456
...
> My guess that this is because the
> recursive hash aggregation misbehaves in a self-similar fashion once
> a
> certain tipping point has been reached.
It looks like it might be fairly easy to use HyperLogLog as an
estimator for the recursive step. That should reduce the
overpartitioning, which I believe is the cause of this discontinuity.
It's not clear to me that overpartitioning is a real problem in this
case -- but I think the fact that it's causing confusion is enough
reason to see if we can fix it.
Regards,
Jeff Davis
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Peter Geoghegan | 2020-07-25 20:27:19 | Re: Default setting for enable_hashagg_disk |
| Previous Message | Peter Geoghegan | 2020-07-25 18:05:02 | Re: Default setting for enable_hashagg_disk |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Peter Geoghegan | 2020-07-25 20:27:19 | Re: Default setting for enable_hashagg_disk |
| Previous Message | Peter Geoghegan | 2020-07-25 19:41:42 | Re: hashagg slowdown due to spill changes |