Skip site navigation (1) Skip section navigation (2)

Re: [PATCH] COPY .. COMPRESSED

From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Bruce Momjian <bruce(at)momjian(dot)us>, Peter Eisentraut <peter_e(at)gmx(dot)net>,pgsql-hackers(at)postgresql(dot)org
Subject: Re: [PATCH] COPY .. COMPRESSED
Date: 2013-01-15 22:19:58
Message-ID: 20130115221957.GX16126@tamriel.snowman.net (view raw or flat)
Thread:
Lists: pgsql-hackers
* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> I find the argument that this supports compression-over-the-wire to be
> quite weak, because COPY is only one form of bulk data transfer, and
> one that a lot of applications don't ever use.  If we think we need to
> support transmission compression for ourselves, it ought to be
> integrated at the wire protocol level, not in COPY.

As far as I can tell, COPY is the option which is strongly recommended
for bulk-data-operations.  I can see the use-case for wanting SELECT
results to be compressed, but it strikes me as the 10% case for PG users
rather than the 90% one.  Ditto for COPY vs. large INSERT .. VALUES.

Compressing every small packet seems like it'd be overkill and might
surprise people by actually reducing performance in the case of lots of
small requests.

It also strikes me as a bit silly to do something like:

zcat myfile.gz | \
psql -Z -c "COPY mytable FROM STDIN;"

> Just to not look like I'm rejecting stuff without proposing
> alternatives, here is an idea about a backwards-compatible design for
> doing that: we could add an option that can be set in the connection
> request packet.  Say, "transmission_compression = gzip".

Alright, do we want/need to support multiple options there?  What do
people think we should support?  Any other particular details or issues
that come to mind with such an implementation?

I'm willing to work through that if it's the route everyone agrees with.

	Thanks,

		Stephen

In response to

Responses

pgsql-hackers by date

Next:From: Bruce MomjianDate: 2013-01-15 22:22:47
Subject: Re: [PATCH] COPY .. COMPRESSED
Previous:From: Bruce MomjianDate: 2013-01-15 22:14:19
Subject: Parallel query execution

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group