Skip site navigation (1) Skip section navigation (2)

Re: multiline CSV fields

From: Kris Jurka <books(at)ejurka(dot)com>
To: Greg Stark <gsstark(at)mit(dot)edu>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: multiline CSV fields
Date: 2004-11-30 12:27:40
Message-ID: Pine.BSO.4.56.0411300718230.29983@leary.csoft.net (view raw or flat)
Thread:
Lists: pgsql-hackerspgsql-patches

On Tue, 30 Nov 2004, Greg Stark wrote:

> 
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> 
> > The advantage of having it in COPY is that it can be done serverside
> > direct from the file system. For massive bulk loads that might be a
> > plus, although I don't know what the protocol+socket overhead is.
> 
> Actually even if you use client-side COPY it's *still* more efficient than any
> more general client-side alternative.

The idea would be to still use COPY just from a program that did 
additional processing, not as direct SQL.

> As Tom pointed out to me a while back, neither the protocol nor libpq allow
> for having multiple queries in flight simultaneously. That makes it impossible
> to stream large quantities of data to the server efficiently. Each record
> requires a round-trip and context switch overhead.

Multiplexing queries is different than having multiple queries in flight.  
You can have multiple queries on the wire now, they are just processed 
sequentially.

> In an ideal world the client should be able to queue up enough records to fill
> the socket buffers and allow the kernel to switch to a more batch oriented
> context switch mode where the server can process large numbers of records
> before switching back to the client. Ideally this would apply to any kind of
> query execution.

This is possible now with the V3 protocol (and used by the JDBC driver).  
For an executeBatch() on a PreparedStatement, the driver sends a parse
message, then any number of bind/execute pairs, but with a buffered stream
nothing happens until a Sync message is sent and the stream is flushed.  
Then it collects the results of all of the executes.

Kris Jurka

In response to

pgsql-hackers by date

Next:From: Vince VielhaberDate: 2004-11-30 12:45:52
Subject: Re: Opinions on Usenet ...
Previous:From: Zeugswetter Andreas DAZ SDDate: 2004-11-30 10:57:34
Subject: Re: Stopgap solution for table-size-estimate updatingproblem

pgsql-patches by date

Next:From: Andrew DunstanDate: 2004-11-30 13:03:48
Subject: Re: multiline CSV fields
Previous:From: Greg StarkDate: 2004-11-30 06:48:35
Subject: Re: multiline CSV fields

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group