Re: Add LZ4 compression in pg_dump

From: gkokolatos(at)pm(dot)me
To: Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com>
Cc: Michael Paquier <michael(at)paquier(dot)xyz>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Andrew Dunstan <andrew(at)dunslane(dot)net>, Alexander Lakhin <exclusion(at)gmail(dot)com>, Justin Pryzby <pryzby(at)telsasoft(dot)com>, shiy(dot)fnst(at)fujitsu(dot)com, pgsql-hackers(at)lists(dot)postgresql(dot)org, Rachel Heaton <rachelmheaton(at)gmail(dot)com>
Subject: Re: Add LZ4 compression in pg_dump
Date: 2023-05-09 14:12:44
Message-ID: MXfs4NbHzTwNxi2reU91_ixbvE2mlGJMIZExtslKMf5DeTn7uCOG1lQrlrPUe9KjdHVkM8628Fl2eIDjXGkHXmf-ACdzG7v6QNnPkETT42s=@pm.me
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

------- Original Message -------
On Tuesday, May 9th, 2023 at 2:54 PM, Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com> wrote:

>
>
> On 5/9/23 00:10, Michael Paquier wrote:
>
> > On Mon, May 08, 2023 at 08:00:39PM +0200, Tomas Vondra wrote:
> >
> > > The LZ4Stream_write() forgot to move the pointer to the next chunk, so
> > > it was happily decompressing the initial chunk over and over. A bit
> > > embarrassing oversight :-(
> > >
> > > The custom format calls WriteDataToArchiveLZ4(), which was correct.
> > >
> > > The attached patch fixes this for me.
> >
> > Ouch. So this was corrupting the dumps and the compression when
> > trying to write more than two chunks at once, not the decompression
> > steps. That addresses the issue here as well, thanks!
>
>
> Yeah. Thanks for the report, should have been found during review.

Thank you both for looking. A small consolation is that now there are
tests for this case.

Moving on to the other open item for this, please find attached v2
of the patch as requested.

Cheers,
//Georgios

>
>
> regards
>
> --
> Tomas Vondra
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company

Attachment Content-Type Size
v2-0001-Null-terminate-the-output-buffer-of-LZ4Stream_get.patch text/x-patch 1.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jonathan S. Katz 2023-05-09 14:19:46 PostgreSQL 16 Beta 1 release date
Previous Message 盏一 2023-05-09 14:12:28 Re: Proposal for Prototype Implementation to Enhance C/C++ Interoperability in PostgreSQL