Skip site navigation (1) Skip section navigation (2)

Re: copy with compression progress n

From: Andreas Pflug <pgadmin(at)pse-consulting(dot)de>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: copy with compression progress n
Date: 2006-05-31 15:31:13
Message-ID: 447DB6C1.1070800@pse-consulting.de (view raw or flat)
Thread:
Lists: pgsql-hackers
Tom Lane wrote:
> Andreas Pflug <pgadmin(at)pse-consulting(dot)de> writes:
> 
>>The attached patch implements COPY ... WITH [BINARY] COMPRESSION 
>>(compression implies BINARY). The copy data uses bit 17 of the flag 
>>field to identify compressed data.
> 
> 
> I think this is a pretty horrid idea, because it changes pg_lzcompress
> from an unimportant implementation detail into a backup file format
> that we have to support till the end of time.  What happens if, say,
> we need to abandon pg_lzcompress because we find out it has patent
> problems?
> 
> It *might* be tolerable if we used gzip instead,

I used pg_lzcompress because it's present in the backend. I'm fine with 
every other good compression algorithm.

>  but I really don't see
> the argument for doing this inside the server at all: piping to gzip
> seems like a perfectly acceptable solution,

As I said, this hits only if it is possible to pipe the result into gzip 
in a performant way. The issue already arises if psql or any other COPY 
client (slony, pg_dump) is not on the same machine: Network bandwidth 
will limit throughput.

> quite possibly with higher
> performance than doing it all in a single process (which isn't going
> to be able to use more than one CPU).

Which is pretty normal for pgsql.

> I don't see the argument for restricting it to binary only, either.

That's not a restriction, but a result: compressed data is binary. 
Marking it as binary will make it working with older frontends as well, 
as long as they don't try to interpret the data. Actually, all 8.x psql 
versions should work (with COPY STDxx, not \copy).

Do you have a comment about the progress notification and its impact on 
copy to stdout?

Regards,
Andreas

In response to

Responses

pgsql-hackers by date

Next:From: Andreas PflugDate: 2006-05-31 15:37:40
Subject: Re: Possible TODO item: copy to/from pipe
Previous:From: David FetterDate: 2006-05-31 15:28:33
Subject: Re: Possible TODO item: copy to/from pipe

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group