Re: Disable WAL logging to speed up data loading

From: Stephen Frost <sfrost(at)snowman(dot)net>
To: "tsunakawa(dot)takay(at)fujitsu(dot)com" <tsunakawa(dot)takay(at)fujitsu(dot)com>
Cc: 'David Steele' <david(at)pgmasters(dot)net>, "osumi(dot)takamichi(at)fujitsu(dot)com" <osumi(dot)takamichi(at)fujitsu(dot)com>, 'Kyotaro Horiguchi' <horikyota(dot)ntt(at)gmail(dot)com>, "sawada(dot)mshk(at)gmail(dot)com" <sawada(dot)mshk(at)gmail(dot)com>, "robertmhaas(at)gmail(dot)com" <robertmhaas(at)gmail(dot)com>, "masao(dot)fujii(at)oss(dot)nttdata(dot)com" <masao(dot)fujii(at)oss(dot)nttdata(dot)com>, "laurenz(dot)albe(at)cybertec(dot)at" <laurenz(dot)albe(at)cybertec(dot)at>, "ashutosh(dot)bapat(dot)oss(at)gmail(dot)com" <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>, "pgsql-hackers(at)lists(dot)postgresql(dot)org" <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: Disable WAL logging to speed up data loading
Date: 2021-03-22 13:46:04
Message-ID: 20210322134603.GR20766@tamriel.snowman.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Greetings,

* tsunakawa(dot)takay(at)fujitsu(dot)com (tsunakawa(dot)takay(at)fujitsu(dot)com) wrote:
> From: Stephen Frost <sfrost(at)snowman(dot)net>
> > The argument here seems to stem from the idea that issueing a 'TRUNCATE'
> > inside the transaction before starting the 'COPY' command is 'too hard'.
>
> > I could be wrong and perhaps opinions will change in the future, but it really
> > doesn't seem like the there's much support for adding a new WAL level just to
> > avoid doing the TRUNCATE. Appealing to what other database systems
> > support can be helpful in some cases but that's usually when we're looking at a
> > new capability which multiple other systems have implemented. This isn't
> > actually a new capability at all- the WAL level that you're looking for already
> > exists and already gives the optimization you're looking for, as long as you issue
> > a TRUNCATE at the start of the transaction.
>
> No, we can't ask using TRUNCATE because the user wants to add data to a table.

First- what are you expecting would actually happen during crash
recovery in this specific case with your proposed new WAL level?

Second, use partitioning, or unlogged tables (with the patch discussed
elsewhere to allow flipping them to logged without writing the entire
thing to WAL).

Thanks,

Stephen

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Robert Haas 2021-03-22 14:05:03 Re: New IndexAM API controlling index vacuum strategies
Previous Message Dean Rasheed 2021-03-22 13:43:53 Re: pgbench - add pseudo-random permutation function