From:
Stephen Frost <sfrost(at)snowman(dot)net>
To:
Claudio Freire <klaussfreire(at)gmail(dot)com>
Cc:
Bruce Momjian <bruce(at)momjian(dot)us>,Gavin Flower <GavinFlower(at)archidevsys(dot)co(dot)nz>,PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject:
Re: Parallel query execution
Date:
2013-01-16 03:55:13
Message-ID:
20130116035513.GE16126@tamriel.snowman.net (view raw or flat )
Thread:
2013-01-15 22:14:19 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-15 22:33:16 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-15 22:39:10 from Peter Geoghegan <peter(at)2ndquadrant(dot)com>
2013-01-15 22:45:15 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-15 22:53:29 from Simon Riggs <simon(at)2ndQuadrant(dot)com>
2013-01-15 22:55:22 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-15 23:01:04 from Simon Riggs <simon(at)2ndQuadrant(dot)com>
2013-01-15 23:03:33 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 11:52:43 from Magnus Hagander <magnus(at)hagander(dot)net>
2013-01-16 13:11:06 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-16 17:16:27 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 17:50:07 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-16 17:56:01 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 18:00:22 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-16 05:35:11 from Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
2013-01-15 23:03:50 from Gavin Flower <GavinFlower(at)archidevsys(dot)co(dot)nz>
2013-01-15 23:08:47 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-15 23:15:57 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-15 23:19:27 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 02:36:55 from Claudio Freire <klaussfreire(at)gmail(dot)com>
2013-01-16 03:13:33 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 03:16:56 from Claudio Freire <klaussfreire(at)gmail(dot)com>
2013-01-16 03:55:13 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 04:47:21 from Claudio Freire <klaussfreire(at)gmail(dot)com>
2013-01-16 13:33:54 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 15:23:14 from Claudio Freire <klaussfreire(at)gmail(dot)com>
2013-01-16 03:56:52 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-16 03:59:35 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 04:22:42 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-16 04:28:18 from Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
2013-01-16 04:32:54 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 04:37:28 from Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
2013-01-16 17:11:42 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 20:07:14 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 21:06:51 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2013-01-16 21:18:38 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 21:57:01 from "Dickson S(dot) Guedes" <listas(at)guedesoft(dot)net>
2013-01-16 22:00:54 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-17 03:32:28 from Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
2013-01-16 04:48:29 from Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>
2013-01-16 07:07:29 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-16 08:21:18 from Abhijit Menon-Sen <ams(at)2ndQuadrant(dot)com>
2013-01-16 12:08:27 from Magnus Hagander <magnus(at)hagander(dot)net>
2013-01-16 13:18:17 from Abhijit Menon-Sen <ams(at)2ndQuadrant(dot)com>
2013-01-16 13:43:18 from Boszormenyi Zoltan <zb(at)cybertec(dot)at>
2013-01-21 23:23:07 from Phil Sorber <phil(at)omniti(dot)com>
2013-01-22 00:31:12 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-22 07:29:29 from Magnus Hagander <magnus(at)hagander(dot)net>
2013-01-16 15:56:14 from Noah Misch <noah(at)leadboat(dot)com>
2013-01-16 12:12:00 from Simon Riggs <simon(at)2ndquadrant(dot)com>
2013-01-16 12:18:04 from Magnus Hagander <magnus(at)hagander(dot)net>
2013-01-16 13:58:12 from Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>
2013-01-16 14:02:45 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 14:10:54 from Abhijit Menon-Sen <ams(at)2ndQuadrant(dot)com>
2013-01-16 14:17:54 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 19:28:05 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-16 19:55:15 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-16 20:13:50 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-16 20:45:46 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-17 05:01:26 from Peter Eisentraut <peter_e(at)gmx(dot)net>
2013-01-17 06:06:13 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-17 01:34:41 from Simon Riggs <simon(at)2ndQuadrant(dot)com>
2013-01-17 03:40:07 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-17 04:48:59 from Abhijit Menon-Sen <ams(at)2ndQuadrant(dot)com>
2013-01-17 06:52:53 from Craig Ringer <craig(at)2ndQuadrant(dot)com>
2013-01-17 07:05:05 from Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
2013-01-17 07:15:52 from Abhijit Menon-Sen <ams(at)2ndQuadrant(dot)com>
2013-01-17 08:43:51 from Magnus Hagander <magnus(at)hagander(dot)net>
2013-01-17 13:17:07 from Craig Ringer <craig(at)2ndQuadrant(dot)com>
2013-01-17 14:54:43 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-16 14:15:44 from Craig Ringer <craig(at)2ndQuadrant(dot)com>
2013-01-16 14:16:33 from Craig Ringer <craig(at)2ndQuadrant(dot)com>
2013-01-16 17:36:10 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-16 22:01:12 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-17 04:38:51 from Craig Ringer <craig(at)2ndQuadrant(dot)com>
2013-01-17 07:19:08 from Pavan Deolasee <pavan(dot)deolasee(at)gmail(dot)com>
2013-01-17 09:22:48 from Magnus Hagander <magnus(at)hagander(dot)net>
2013-01-19 22:21:29 from Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
2013-01-20 18:42:45 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-20 19:39:05 from Simon Riggs <simon(at)2ndQuadrant(dot)com>
2013-01-20 20:18:34 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2013-01-20 21:57:57 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-21 00:07:21 from Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
2013-01-21 02:37:32 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-21 03:04:22 from Andrew Dunstan <andrew(at)dunslane(dot)net>
2013-01-21 03:07:07 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-21 03:11:34 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-21 22:48:38 from Dimitri Fontaine <dimitri(at)2ndQuadrant(dot)fr>
2013-01-22 01:23:00 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-22 01:47:12 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-22 02:27:55 from Phil Sorber <phil(at)omniti(dot)com>
2013-01-22 05:24:18 from Pavan Deolasee <pavan(dot)deolasee(at)gmail(dot)com>
2013-01-22 06:15:38 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-22 09:35:51 from Dimitri Fontaine <dimitri(at)2ndQuadrant(dot)fr>
2013-01-22 17:09:02 from Gavin Flower <GavinFlower(at)archidevsys(dot)co(dot)nz>
2013-01-22 12:44:41 from Pavan Deolasee <pavan(dot)deolasee(at)gmail(dot)com>
2013-01-22 13:16:40 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2013-01-22 14:22:04 from Andrew Dunstan <andrew(at)dunslane(dot)net>
2013-01-23 16:44:29 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-23 17:08:39 from Andres Freund <andres(at)2ndquadrant(dot)com>
2013-01-23 17:51:15 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-23 18:27:14 from "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
2013-01-23 18:44:03 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-23 20:23:32 from Phil Sorber <phil(at)omniti(dot)com>
2013-01-23 20:31:00 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-23 20:48:50 from Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
2013-01-23 21:27:58 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-23 21:47:58 from Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
2013-01-24 11:33:37 from Amit Kapila <amit(dot)kapila(at)huawei(dot)com>
2013-01-24 09:13:39 from Pavan Deolasee <pavan(dot)deolasee(at)gmail(dot)com>
2013-01-22 03:33:49 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-22 02:22:47 from Phil Sorber <phil(at)omniti(dot)com>
2013-01-22 13:15:28 from Dimitri Fontaine <dimitri(at)2ndQuadrant(dot)fr>
2013-01-21 07:48:45 from Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
2013-01-21 18:54:33 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-21 19:04:14 from Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
2013-01-21 19:09:57 from Josh Berkus <josh(at)agliodbs(dot)com>
2013-01-21 19:39:20 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-23 17:19:08 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 09:37:04 from Daniel Farina <daniel(at)heroku(dot)com>
2013-01-16 13:42:29 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 16:02:10 from Noah Misch <noah(at)leadboat(dot)com>
2013-01-16 12:05:36 from Andres Freund <andres(at)2ndquadrant(dot)com>
2013-01-16 12:53:12 from Robert Haas <robertmhaas(at)gmail(dot)com>
2013-01-16 13:38:08 from Stephen Frost <sfrost(at)snowman(dot)net>
2013-01-16 17:13:51 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 14:05:39 from Andrew Dunstan <andrew(at)dunslane(dot)net>
2013-01-16 17:20:24 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-16 17:29:16 from Andrew Dunstan <andrew(at)dunslane(dot)net>
2013-01-17 01:04:05 from Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
2013-01-17 02:44:53 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-17 02:56:21 from Claudio Freire <klaussfreire(at)gmail(dot)com>
2013-01-17 03:29:24 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-17 01:04:05 from Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
2013-01-17 02:42:04 from Claudio Freire <klaussfreire(at)gmail(dot)com>
2013-01-16 00:11:20 from Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
2013-01-16 02:29:01 from Bruce Momjian <bruce(at)momjian(dot)us>
2013-01-24 22:34:49 from Paul Ramsey <pramsey(at)cleverelephant(dot)ca>
2013-01-24 23:17:18 from Bruce Momjian <bruce(at)momjian(dot)us>
Lists:
pgsql-hackers
* Claudio Freire (klaussfreire(at)gmail(dot)com) wrote:
> On Wed, Jan 16, 2013 at 12:13 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> > Sequentially scanning the *same* data over and over is certainly
> > counterprouctive. Synchroscans fixed that, yes. That's not what we're
> > talking about though- we're talking about scanning and processing
> > independent sets of data using multiple processes.
>
> I don't see the difference. Blocks are blocks (unless they're cached).
Not quite. Having to go out to the kernel isn't free. Additionally,
the seq scans used to pollute our shared buffers prior to
synch-scanning, which didn't help things.
> > It's certainly
> > possible that in some cases that won't be as good
>
> If memory serves me correctly (and it does, I suffered it a lot), the
> performance hit is quite considerable. Enough to make it "a lot worse"
> rather than "not as good".
I feel like we must not be communicating very well.
If the CPU is pegged at 100% and the I/O system is at 20%, adding
another CPU at 100% will bring the I/O load up to 40% and you're now
processing data twice as fast overall. If you're running a single CPU
at 20% and your I/O system is at 100%, then adding another CPU isn't
going to help and may even degrade performance by causing problems for
the I/O system. The goal of the optimizer will be to model the plan to
account for exactly that, as best it can.
> > but there will be
> > quite a few cases where it's much, much better.
>
> Just cached segments.
No, certainly not just cached segments. Any situation where the CPU is
the bottleneck.
Thanks,
Stephen
In response to
Responses
pgsql-hackers by date
Next :From: Josh BerkusDate: 2013-01-16 03:56:52
Subject : Re: Parallel query execution
Previous :From : Stephen FrostDate : 2013-01-16 03:47:57
Subject : Re: log_lock_waits to identify transaction's relation