Skip site navigation (1) Skip section navigation (2)

Re: copy with compression progress n

From: Hannu Krosing <hannu(at)skype(dot)net>
To: Andreas Pflug <pgadmin(at)pse-consulting(dot)de>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>,PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: copy with compression progress n
Date: 2006-06-01 05:23:31
Message-ID: 1149139411.3839.3.camel@localhost.localdomain (view raw or flat)
Thread:
Lists: pgsql-hackers
Ühel kenal päeval, K, 2006-05-31 kell 17:31, kirjutas Andreas Pflug:
> Tom Lane wrote:
> > Andreas Pflug <pgadmin(at)pse-consulting(dot)de> writes:
> > 
> >>The attached patch implements COPY ... WITH [BINARY] COMPRESSION 
> >>(compression implies BINARY). The copy data uses bit 17 of the flag 
> >>field to identify compressed data.
> > 
> > 
> > I think this is a pretty horrid idea, because it changes pg_lzcompress
> > from an unimportant implementation detail into a backup file format
> > that we have to support till the end of time.  What happens if, say,
> > we need to abandon pg_lzcompress because we find out it has patent
> > problems?
> > 
> > It *might* be tolerable if we used gzip instead,
> 
> I used pg_lzcompress because it's present in the backend. I'm fine with 
> every other good compression algorithm.
> 
> >  but I really don't see
> > the argument for doing this inside the server at all: piping to gzip
> > seems like a perfectly acceptable solution,
> 
> As I said, this hits only if it is possible to pipe the result into gzip 
> in a performant way. The issue already arises if psql or any other COPY 
> client (slony, pg_dump) is not on the same machine: Network bandwidth 
> will limit throughput.

Maybe make up a way to pipe COPY result through some external process
(like gzip) on the server side without having shell access there.

To make it secure, the external process should probably be run from a
hardwired directory via chroot.

-- 
----------------
Hannu Krosing
Database Architect
Skype Technologies OÜ
Akadeemia tee 21 F, Tallinn, 12618, Estonia

Skype me:  callto:hkrosing
Get Skype for free:  http://www.skype.com



In response to

pgsql-hackers by date

Next:From: Martijn van OosterhoutDate: 2006-06-01 05:42:23
Subject: Re: Possible TODO item: copy to/from pipe
Previous:From: Bruce MomjianDate: 2006-06-01 05:12:43
Subject: Re: [HACKERS] psql \copy warning

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group