Re: Support for pg_receivexlog --format=plain|tar

From: Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
To: Magnus Hagander <magnus(at)hagander(dot)net>
Cc: PostgreSQL mailing lists <pgsql-hackers(at)postgresql(dot)org>, David Steele <david(at)pgmasters(dot)net>
Subject: Re: Support for pg_receivexlog --format=plain|tar
Date: 2016-12-28 05:58:27
Message-ID: CAB7nPqRfabqm4qqgkvVznO24HDYazNaWWTK9_+Jzp=sgnYwN_Q@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Wed, Dec 28, 2016 at 3:12 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Tue, Dec 27, 2016 at 1:16 PM, Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
> wrote:
>> On Tue, Dec 27, 2016 at 6:34 PM, Magnus Hagander <magnus(at)hagander(dot)net>
>> wrote:
>> > On Tue, Dec 27, 2016 at 2:23 AM, Michael Paquier
>> > <michael(dot)paquier(at)gmail(dot)com>
>> > wrote:
>> >> Magnus, you have mentioned me as well that you had a couple of ideas
>> >> on the matter, feel free to jump in and let's mix our thoughts!
>> >
>> >
>> > Yeah, I've been wondering what the actual usecase is here :)
>>
>> There is value to compress segments finishing with trailing zeros,
>> even if they are not the species with the highest representation in
>> the WAL archive.
>
> Agreed on that part -- that's the value in compression though, and not
> necessarily the TAR format itself.
>
> Is there any value of the TAR format *without* compression in your scenario?

Hm. I cannot think of one.

>> > I can see the point of being able to compress the individual segments
>> > directly in pg_receivexlog in smaller systems though, without the need
>> > to
>> > rely on an external compression program as well. But in that case, is
>> > there
>> > any reason we need to wrap it in a tarfile, and can't just write it to
>> > <segment>.gz natively?
>>
>> You mean having a --compress=0|9 option that creates individual gz
>> files for each segment? Definitely we could just do that. It would be
>
> Yes, that's what I meant.

OK, I have bitten the bullet and attached is a patch to add just
--compress=0|9. So there is one .gz file generated per segment, and
things are rather straight-forward. Adding tests is unfortunately not
in scope as not all builds are compiled with libz.

A couple of things to be aware of though:
- gzopen, gzwrite and gzclose are used to handle the gz files. That's
unconsistent with the tar method that is based on mainly deflate() and
more low level routines.
- I have switched the directory method to use a file pointer instead
of a file descriptor as gzwrite returns int as the number of
uncompressed bytes written.
- history and timeline files are gzip'd as well. Perhaps we don't want
to do that.

What do you think about this approach? I'll add that to the next CF.
--
Michael

Attachment Content-Type Size
receivexlog-gzip-v1.patch invalid/octet-stream 11.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Michael Paquier 2016-12-28 06:07:52 Re: Hooks
Previous Message David Fetter 2016-12-28 05:14:35 Re: Hooks