Skip site navigation (1) Skip section navigation (2)

Re: Benchmark Data requested --- pgloader CE design ideas

From: Greg Smith <gsmith(at)gregsmith(dot)com>
To: pgsql-performance(at)postgresql(dot)org
Subject: Re: Benchmark Data requested --- pgloader CE design ideas
Date: 2008-02-07 17:06:42
Message-ID: Pine.GSO.4.64.0802071157060.27404@westnet.com (view raw or flat)
Thread:
Lists: pgsql-performance
On Thu, 7 Feb 2008, Dimitri Fontaine wrote:

> I was thinking of not even reading the file content from the controller
> thread, just decide splitting points in bytes (0..ST_SIZE/4 -
> ST_SIZE/4+1..2*ST_SIZE/4 etc) and let the reading thread fine-tune by
> beginning to process input after having read first newline, etc.

The problem I was pointing out is that if chunk#2 moved foward a few bytes 
before it started reading in search of a newline, how will chunk#1 know 
that it's supposed to read up to that further point?  You have to stop #1 
from reading further when it catches up with where #2 started.  Since the 
start of #2 is fuzzy until some reading is done, what you're describing 
will need #2 to send some feedback to #1 after they've both started, and 
that sounds bad to me.  I like designs where the boundaries between 
threads are clearly defined before any of them start and none of them ever 
talk to the others.

> In both cases, maybe it would also be needed for pgloader to be able to have a
> separate thread for COPYing the buffer to the server, allowing it to continue
> preparing next buffer in the meantime?

That sounds like a V2.0 design to me.  I'd only chase after that level of 
complexity if profiling suggests that's where the bottleneck really is.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

In response to

Responses

pgsql-performance by date

Next:From: MatthewDate: 2008-02-07 17:11:10
Subject: Re: Benchmark Data requested --- pgloader CE design ideas
Previous:From: Markus BertheauDate: 2008-02-07 14:51:57
Subject: Index Scan Backward + check additional condition before heap access

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group