From:
Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To:
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc:
Simon Riggs <simon(at)2ndquadrant(dot)com>, "David E(dot)Wheeler" <david(at)kineticode(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject:
Re: Review: listagg aggregate
Date:
2010-01-25 15:04:59
Message-ID:
162867791001250704i19d61591y6e7fc504d6bdc9d6@mail.gmail.com (view raw or flat )
Thread:
2010-01-22 19:14:12 from David E(dot) Wheeler <david(at)kineticode(dot)com>
2010-01-24 09:19:41 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-24 16:57:36 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-24 18:45:00 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-25 05:29:38 from Scott Bailey <artacus(at)comcast(dot)net>
2010-01-25 10:09:33 from Peter Eisentraut <peter_e(at)gmx(dot)net>
2010-01-25 10:34:59 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-25 13:57:10 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-25 14:12:37 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-25 19:29:39 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-26 06:59:35 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-25 14:56:06 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-26 18:14:47 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-27 15:58:17 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 00:45:07 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-28 02:47:07 from Takahiro Itagaki <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
2010-01-28 03:28:31 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-28 08:38:10 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 08:40:01 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 08:59:11 from Takahiro Itagaki <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
2010-01-28 13:53:26 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-28 14:01:19 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 14:17:00 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-28 15:32:33 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 17:08:41 from Hitoshi Harada <umi(dot)tanuki(at)gmail(dot)com>
2010-01-28 17:17:50 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 17:29:04 from Marko Tiikkaja <marko(dot)tiikkaja(at)cs(dot)helsinki(dot)fi>
2010-01-28 17:30:30 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-28 17:20:45 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-28 15:56:45 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-28 16:01:23 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-28 17:12:50 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 17:16:59 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-28 17:19:59 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 17:24:33 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-29 07:43:48 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-29 18:43:04 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-29 18:45:09 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-29 18:46:50 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-29 18:47:42 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-02-01 03:29:37 from Takahiro Itagaki <itagaki(dot)takahiro(at)oss(dot)ntt(dot)co(dot)jp>
2010-02-01 05:07:52 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-02-01 12:11:27 from Thom Brown <thombrown(at)gmail(dot)com>
2010-02-01 13:40:50 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-02-01 13:51:42 from Thom Brown <thombrown(at)gmail(dot)com>
2010-02-01 15:40:37 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-28 18:57:50 from Greg Stark <stark(at)mit(dot)edu>
2010-01-28 08:37:46 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-28 16:02:21 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-28 17:14:37 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-25 19:27:57 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-25 20:14:22 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-26 07:14:13 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-26 11:08:16 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-26 11:23:58 from Alastair Turner <bell(at)ctrlf5(dot)co(dot)za>
2010-01-26 11:44:34 from Alastair Turner <bell(at)ctrlf5(dot)co(dot)za>
2010-01-26 12:03:25 from Peter Eisentraut <peter_e(at)gmx(dot)net>
2010-01-26 15:24:09 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-26 17:36:13 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-26 18:04:23 from "David E(dot) Wheeler" <david(at)kineticode(dot)com>
2010-01-26 18:09:05 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-26 18:15:15 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-26 18:38:37 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-27 22:09:14 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-26 14:15:35 from Robert Haas <robertmhaas(at)gmail(dot)com>
2010-01-26 14:56:07 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-26 16:03:57 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-26 16:15:36 from "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
2010-01-26 17:19:17 from "Erik Rijkers" <er(at)xs4all(dot)nl>
2010-01-26 17:39:19 from "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
2010-01-26 19:02:53 from Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
2010-01-27 03:46:32 from Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
2010-01-27 10:05:53 from Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com>
2010-01-27 10:46:36 from Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
2010-01-27 11:37:43 from Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com>
2010-01-27 13:11:45 from Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
2010-01-27 16:38:00 from Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com>
2010-01-28 00:32:58 from "Erik Rijkers" <er(at)xs4all(dot)nl>
2010-01-28 01:09:14 from Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
2010-01-24 18:02:25 from Simon Riggs <simon(at)2ndQuadrant(dot)com>
2010-01-24 18:40:16 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2010-01-24 18:43:54 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2010-01-25 15:04:59 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Lists:
pgsql-hackers
2010/1/24 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>:
> Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
>> On Fri, 2010-01-22 at 11:14 -0800, David E.Wheeler wrote:
>>> No performance issues
>
>> ISTM that this class of function is inherently dangerous performance
>> wise.
>
>> * It looks incredibly easy to construct enormous lists. We should test
>> the explosion limit of this to see how it is handled. Perhaps we need
>> some parameter limits to control that, depending upon results.
>
>> * Optimizer doesn't consider whether the result type of an aggregate get
>> bigger as the aggregate processes more rows. If we're adding this
>> function we should give some thought in that area also, or at least a
>> comment to note that it can and will cause the optimizer problems in
>> complex queries.
>
> We have that problem already with array_agg(), and I don't recall many
> complaints about it. It might be worth worrying about at some point,
> but I don't think it's reasonable to insist that it be fixed before
> any more such aggregates are created.
>
> I agree that testing-to-failure would be a good idea just to be sure it
> fails cleanly.
postgres=# \timing
Timing is on.
postgres=# select
pg_size_pretty(length(string_agg('012345678901234567890'::text,',')))
from generate_series(1,10000000) g(i);
pg_size_pretty
----------------
210 MB
(1 row)
Time: 5831,218 ms
postgres=# select
pg_size_pretty(length(string_agg('012345678901234567890'::text,',')))
from generate_series(1,50000000) g(i);
^[^[ERROR: out of memory
DETAIL: Cannot enlarge string buffer containing 1073741812 bytes by
21 more bytes.
postgres=#
I thing, so 210 MB is more then is necessary :)
Regards
Pavel Stehule
>
> regards, tom lane
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
In response to
pgsql-hackers by date
Next :From: Cédric VillemainDate: 2010-01-25 15:46:26
Subject : Re: MySQL-ism help patch for psql
Previous :From : Pavel StehuleDate : 2010-01-25 14:56:06
Subject : Re: Review: listagg aggregate