Re: Benchmark Data requested --- pgloader CE design ideas

Le mercredi 06 février 2008, Greg Smith a écrit :
> If I'm loading a TB file, odds are good I can split that into 4 or more
> vertical pieces (say rows 1-25%, 25-50%, 50-75%, 75-100%), start 4 loaders
> at once, and get way more than 1 disk worth of throughput reading.

pgloader already supports starting at any input file line number, and limit
itself to any number of reads:

-C COUNT, --count=COUNT
number of input lines to process
-F FROMCOUNT, --from=FROMCOUNT
number of input lines to skip

So you could already launch 4 pgloader processes with the same configuration
fine but different command lines arguments. It there's interest/demand, it's
easy enough for me to add those parameters as file configuration knobs too.

Still you have to pay for client to server communication instead of having the
backend read the file locally, but now maybe we begin to compete?

Regards,
--
dim