Re: multiline CSV fields

On Tue, 30 Nov 2004, Greg Stark wrote:

>
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
> > The advantage of having it in COPY is that it can be done serverside
> > direct from the file system. For massive bulk loads that might be a
> > plus, although I don't know what the protocol+socket overhead is.
>
> Actually even if you use client-side COPY it's *still* more efficient than any
> more general client-side alternative.

The idea would be to still use COPY just from a program that did
additional processing, not as direct SQL.

> As Tom pointed out to me a while back, neither the protocol nor libpq allow
> for having multiple queries in flight simultaneously. That makes it impossible
> to stream large quantities of data to the server efficiently. Each record
> requires a round-trip and context switch overhead.

Multiplexing queries is different than having multiple queries in flight.
You can have multiple queries on the wire now, they are just processed
sequentially.

> In an ideal world the client should be able to queue up enough records to fill
> the socket buffers and allow the kernel to switch to a more batch oriented
> context switch mode where the server can process large numbers of records
> before switching back to the client. Ideally this would apply to any kind of
> query execution.

This is possible now with the V3 protocol (and used by the JDBC driver).
For an executeBatch() on a PreparedStatement, the driver sends a parse
message, then any number of bind/execute pairs, but with a buffered stream
nothing happens until a Sync message is sent and the stream is flushed.
Then it collects the results of all of the executes.

Kris Jurka