Re: Producer/Consumer Issues in the COPY across network

On Tue, 2008-02-26 at 12:29 +0100, Martijn van Oosterhout wrote:

> > When we're running a COPY over a high latency link then network time is
> > going to become dominant, so potentially, running COPY asynchronously
> > might help performance for loads or initial Slony configuration. This is
> > potentially more important on Slony where we do both a PQgetCopyData()
> > and PQputCopyData() in a tight loop.
>
> When you check the packets being sent, are you showing only one record
> being sent per packet? If so, there's your problem.

I've not inspected the packet flow. It seemed easier to ask.

> > I also note that PQgetCopyData always returns just one row. Is there an
> > underlying buffering between the protocol (which always sends one
> > message per row) and libpq (which is one call per row)? It seems
> > possible for us to request a number of rows from the server up to a
> > preferred total transfer size.
>
> AIUI the server merely streams the rows to you, the client doesn't get
> to say how many :)

Right, but presumably we generate a new message per PQgetCopyData()
request? So my presumption is we need to wait for that to be generated
each time?

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com