From: | Simon Riggs <simon(at)2ndQuadrant(dot)com> |
---|---|
To: | Robert Haas <robertmhaas(at)gmail(dot)com> |
Cc: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Postgres Hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: BufferAccessStrategy for bulk insert |
Date: | 2008-10-31 10:53:01 |
Message-ID: | 1225450381.3971.517.camel@ebony.2ndQuadrant |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Thu, 2008-10-30 at 22:46 -0400, Robert Haas wrote:
> > You should try profiling the patch. You can count the invocations of the
> > buffer access routines to check its all working in the right ratios.
>
> *goes and learns how to do profile PostgreSQL*
>
> OK, that was a good suggestion. It looks like part of my problem here
> is that I didn't put the CREATE TABLE and the COPY into the same
> transaction. As a result, a lot of time was spent on XLogInsert.
> Modified the test case, new profiling results attached.
The CPU time in XLogInsert can be confusing. The WAL writes can make
COPY I/O bound and so any savings on CPU may have been masked in the
earlier tests.
Patched profile shows we can still save a further 20% by writing data
block-at-a-time. That's more complex because we'd need to buffer the
index inserts also, or it would optimise only for the no-index (initial
load) case. So I think this is definitely enough for this release.
Using the buffer access strategy is going to be a big win for people
running large data loads in production and it will also help with people
running parallel load tasks (e.g. Dimitri's pg_loader). That effect is
more subtle and harder to measure, but it's an important consideration.
Thanks very much for finishing the patch in time for commitfest.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
From | Date | Subject | |
---|---|---|---|
Next Message | Zdenek Kotala | 2008-10-31 11:25:59 | Re: PG_PAGE_LAYOUT_VERSION 5 - time for change |
Previous Message | Florian Weimer | 2008-10-31 10:21:21 | Re: Block-level CRC checks |