Skip site navigation (1) Skip section navigation (2)

Re: Producer/Consumer Issues in the COPY across network

From: Simon Riggs <simon(at)2ndquadrant(dot)com>
To: Martijn van Oosterhout <kleptog(at)svana(dot)org>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Producer/Consumer Issues in the COPY across network
Date: 2008-02-28 01:57:49
Message-ID: 1204163869.4252.758.camel@ebony.site (view raw or flat)
Thread:
Lists: pgsql-hackers
On Tue, 2008-02-26 at 12:29 +0100, Martijn van Oosterhout wrote:

> > When we're running a COPY over a high latency link then network time is
> > going to become dominant, so potentially, running COPY asynchronously
> > might help performance for loads or initial Slony configuration. This is
> > potentially more important on Slony where we do both a PQgetCopyData()
> > and PQputCopyData() in a tight loop.
> 
> When you check the packets being sent, are you showing only one record
> being sent per packet? If so, there's your problem.

I've not inspected the packet flow. It seemed easier to ask.

> > I also note that PQgetCopyData always returns just one row. Is there an
> > underlying buffering between the protocol (which always sends one
> > message per row) and libpq (which is one call per row)? It seems
> > possible for us to request a number of rows from the server up to a
> > preferred total transfer size.
> 
> AIUI the server merely streams the rows to you, the client doesn't get
> to say how many :)

Right, but presumably we generate a new message per PQgetCopyData()
request? So my presumption is we need to wait for that to be generated
each time?

-- 
  Simon Riggs
  2ndQuadrant  http://www.2ndQuadrant.com 


In response to

Responses

pgsql-hackers by date

Next:From: Babu, Gabriel SureshDate: 2008-02-28 07:02:28
Subject: ES7000 Windows 2003 server 64bit processor
Previous:From: Tom LaneDate: 2008-02-28 00:53:48
Subject: Buildfarm member gypsy_moth seems not to like alignment patch

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group