RE: Disable WAL logging to speed up data loading

From: "osumi(dot)takamichi(at)fujitsu(dot)com" <osumi(dot)takamichi(at)fujitsu(dot)com>
To: 'Fujii Masao' <masao(dot)fujii(at)oss(dot)nttdata(dot)com>, 'Ashutosh Bapat' <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>, "tsunakawa(dot)takay(at)fujitsu(dot)com" <tsunakawa(dot)takay(at)fujitsu(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: RE: Disable WAL logging to speed up data loading
Date: 2020-10-01 03:04:38
Message-ID: OSBPR01MB4888372E2590774B1311F4B1ED300@OSBPR01MB4888.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello.

> >> Can they use a database with all unlogged tables?
> > Unfortunately, no. They want to switch a cluster condition to "without WAL
> logging"
> > only when they execute night bulk loading for their data warehouse.
> > In other words, they would like to keep their other usual operations with WAL.
> > In addition, using all tables as unlogged brings about the risk to
> > lose data warehouse's data caused by an unexpected server crash or power
> outage.
>
> But the same issue can happen even in the proposed approach because
> Tsunakawa-san explains as follows?
Sorry, my last expression about "to lose data" was not precise.

When unlogged tables are used and the server is crashed unexpectedly,
all data after the last backup is truncated without WAL, right ?
In this case, sequential commands for unlogged tables disappear.
On the other hand, wal_level='none' can be changed to 'minimal' for example.
Therefore, such other sequential operations after the last backup
could be stored in WAL file, when a user applies such a change of wal_level.
I meant this is helpful but didn't this make sense ?

> > The server refuses to start (pg_ctl start fails) after an abnormal shutdown due
> to power outage, pg_ctl's immediate shutdown, etc, showing a straightforward
> message like MySQL.
This is the behavior during wal_level='none'.

> * The user can use all features again if you shut down the server successfully
> after data loading and reset wal_level to a value other than none. He needs to
> take a base backup or rebuild the replication standby after restarting the server.
And, we expect we can change the wal_level even after using wal_level='none'
and can use other features again like I mentioned above. What did you think ?

Regards,
Takamichi Osumi

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Michael Paquier 2020-10-01 03:15:38 Re: Why does PostgresNode.pm set such a low value of max_wal_senders?
Previous Message tsunakawa.takay@fujitsu.com 2020-10-01 02:55:46 RE: [Patch] Optimize dropping of relation buffers using dlist