Re: Add LZ4 compression in pg_dump

------- Original Message -------
On Monday, May 8th, 2023 at 8:20 PM, Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com> wrote:

>
>
>
>
> On 5/8/23 18:19, gkokolatos(at)pm(dot)me wrote:
>
> > ------- Original Message -------
> > On Monday, May 8th, 2023 at 3:16 AM, Tom Lane tgl(at)sss(dot)pgh(dot)pa(dot)us wrote:
> >
> > > I wrote:
> > >
> > > > Michael Paquier michael(at)paquier(dot)xyz writes:
> > > >
> > > > > While testing this patch, I have triggered an error pointing out that
> > > > > the decompression path of LZ4 is broken for table data. I can
> > > > > reproduce that with a dump of the regression database, as of:
> > > > > make installcheck
> > > > > pg_dump --format=d --file=dump_lz4 --compress=lz4 regression
> > >
> > > > Ugh. Reproduced here ... so we need an open item for this.
> > >
> > > BTW, it seems to work with --format=c.
> >
> > Thank you for the extra tests. It seems that exists a gap in the test
> > coverage. Please find a patch attached that is addressing the issue
> > and attempt to provide tests for it.
>
>
> Seems I'm getting messages with a delay - this is mostly the same fix I
> ended up with, not realizing you already posted a fix.

Thank you very much for looking.

> I don't think we need the local "in" variable - the pointer parameter is
> local in the function, so we can modify it directly (with a cast).
> WriteDataToArchiveLZ4 does it that way too.

Sure, patch updated.

> The tests are definitely a good idea.

Thank you.

> I wonder if we should add a
> comment to DEFAULT_IO_BUFFER_SIZE mentioning that if we choose to
> increase the value in the future, we needs to tweak the tests too to use
> more data in order to exercise the buffering etc. Maybe it's obvious?
>

You are right. Added a comment both in the header and in the test.

I hope v2 gets closer to closing the open item for this.

Cheers,
//Georgios

>
> regards
>
> --
> Tomas Vondra
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company