RE: Disable WAL logging to speed up data loading

From: "tsunakawa(dot)takay(at)fujitsu(dot)com" <tsunakawa(dot)takay(at)fujitsu(dot)com>
To: "osumi(dot)takamichi(at)fujitsu(dot)com" <osumi(dot)takamichi(at)fujitsu(dot)com>
Cc: 'Kyotaro Horiguchi' <horikyota(dot)ntt(at)gmail(dot)com>, "sawada(dot)mshk(at)gmail(dot)com" <sawada(dot)mshk(at)gmail(dot)com>, "masao(dot)fujii(at)oss(dot)nttdata(dot)com" <masao(dot)fujii(at)oss(dot)nttdata(dot)com>, "laurenz(dot)albe(at)cybertec(dot)at" <laurenz(dot)albe(at)cybertec(dot)at>, "ashutosh(dot)bapat(dot)oss(at)gmail(dot)com" <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>, "pgsql-hackers(at)lists(dot)postgresql(dot)org" <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: RE: Disable WAL logging to speed up data loading
Date: 2020-12-02 02:21:52
Message-ID: TYAPR01MB29906BC112B9639F4679D92BFEF30@TYAPR01MB2990.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

From: Osumi, Takamichi/大墨 昂道 <osumi(dot)takamichi(at)fujitsu(dot)com>
> I executed each wal_level three times and calculated the average time
> and found that disabling WAL logging reduced about 73 % of the minimal's
> loading speed
> in this test. This speed-up came from the difference of generated WAL sizes.

So, it's 4x speedup when the WAL buffer and/or disk is likely to be saturated. That's nice!

> In this test, to load the data generated more than 10GB of WALs with
> wal_level=minimal
> while wal_level=none emits just 760 bytes of WALs.
>
> I expect this size for none will become smaller when
> I take the modification to filter out the types of WAL which is discussed above.

I don't expect so, because the WAL volume is already very small in this workload (probably only the commit records.)

> Also, I monitored numbers of iostat's 'util' and noticed that
> util's spike to use I/O reduced from twice to once when I changed the level
> from minimal to none, which should be the effect of the patch.

It's a bit mysterious. I thought %util would be constantly 100% when wal_level = minimal. That may be because wal_buffers is large.

Regards
Takayuki Tsunakawa

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Michael Paquier 2020-12-02 03:03:49 Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2
Previous Message Mark Dilger 2020-12-02 02:03:48 Re: room for improvement in amcheck btree checking?