From: | Magnus Hagander <magnus(at)hagander(dot)net> |
---|---|
To: | Michael Paquier <michael(dot)paquier(at)gmail(dot)com> |
Cc: | PostgreSQL mailing lists <pgsql-hackers(at)postgresql(dot)org>, David Steele <david(at)pgmasters(dot)net> |
Subject: | Re: Support for pg_receivexlog --format=plain|tar |
Date: | 2016-12-28 12:31:33 |
Message-ID: | CABUevEw0SPkMt75_C8=Eq-a3aKs9=Q+zay6ZpkS-TnZYir=Juw@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On Wed, Dec 28, 2016 at 6:58 AM, Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
wrote:
> On Wed, Dec 28, 2016 at 3:12 AM, Magnus Hagander <magnus(at)hagander(dot)net>
> wrote:
> > On Tue, Dec 27, 2016 at 1:16 PM, Michael Paquier <
> michael(dot)paquier(at)gmail(dot)com>
> > wrote:
> >> On Tue, Dec 27, 2016 at 6:34 PM, Magnus Hagander <magnus(at)hagander(dot)net>
> >> wrote:
> >> > On Tue, Dec 27, 2016 at 2:23 AM, Michael Paquier
> >> > <michael(dot)paquier(at)gmail(dot)com>
> >> > wrote:
> >> >> Magnus, you have mentioned me as well that you had a couple of ideas
> >> >> on the matter, feel free to jump in and let's mix our thoughts!
> >> >
> >> >
> >> > Yeah, I've been wondering what the actual usecase is here :)
> >>
> >> There is value to compress segments finishing with trailing zeros,
> >> even if they are not the species with the highest representation in
> >> the WAL archive.
> >
> > Agreed on that part -- that's the value in compression though, and not
> > necessarily the TAR format itself.
> >
> > Is there any value of the TAR format *without* compression in your
> scenario?
>
> Hm. I cannot think of one.
>
> >> > I can see the point of being able to compress the individual segments
> >> > directly in pg_receivexlog in smaller systems though, without the need
> >> > to
> >> > rely on an external compression program as well. But in that case, is
> >> > there
> >> > any reason we need to wrap it in a tarfile, and can't just write it to
> >> > <segment>.gz natively?
> >>
> >> You mean having a --compress=0|9 option that creates individual gz
> >> files for each segment? Definitely we could just do that. It would be
> >
> > Yes, that's what I meant.
>
> OK, I have bitten the bullet and attached is a patch to add just
> --compress=0|9. So there is one .gz file generated per segment, and
> things are rather straight-forward. Adding tests is unfortunately not
> in scope as not all builds are compiled with libz.
>
Conditional tests? It probably wouldn't hurt to have them, but that would
be something more generic (like we'd need something to actually validate it
-- but it would make sense to have a test that, with compression enabled,
would verify if the uncompressed file turns out to be exactly 16Mb for
example).
> A couple of things to be aware of though:
> - gzopen, gzwrite and gzclose are used to handle the gz files. That's
> unconsistent with the tar method that is based on mainly deflate() and
> more low level routines.
>
But chosen for simplicity, I assume?
> - I have switched the directory method to use a file pointer instead
> of a file descriptor as gzwrite returns int as the number of
> uncompressed bytes written.
>
I don't really follow that reasoning :) Why does the directory method have
to change to use a filepointer because of that?
> - history and timeline files are gzip'd as well. Perhaps we don't want
> to do that.
>
I think it makes sense to compress everything.
> What do you think about this approach? I'll add that to the next CF.
I haven't reviweed the code in detail but yes, I think this approach is the
right one.
//Magnus
From | Date | Subject | |
---|---|---|---|
Next Message | Anastasia Lubennikova | 2016-12-28 12:37:04 | Re: Vacuum: allow usage of more than 1GB of work mem |
Previous Message | Thomas Munro | 2016-12-28 12:28:54 | Re: Measuring replay lag |