RE: Disable WAL logging to speed up data loading

From: "tsunakawa(dot)takay(at)fujitsu(dot)com" <tsunakawa(dot)takay(at)fujitsu(dot)com>
To: "osumi(dot)takamichi(at)fujitsu(dot)com" <osumi(dot)takamichi(at)fujitsu(dot)com>
Cc: 'Fujii Masao' <masao(dot)fujii(at)oss(dot)nttdata(dot)com>, Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at>, Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>, "ashutosh(dot)bapat(dot)oss(at)gmail(dot)com" <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>, "pgsql-hackers(at)lists(dot)postgresql(dot)org" <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: RE: Disable WAL logging to speed up data loading
Date: 2020-11-19 07:50:46
Message-ID: TYAPR01MB29906AA17E5E04820D1572C7FEE00@TYAPR01MB2990.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

(1)
#define RelationNeedsWAL(relation) \
+ (wal_level != WAL_LEVEL_NONE && \
((relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(XLogIsNeeded() || \
(relation->rd_createSubid == InvalidSubTransactionId && \
- relation->rd_firstRelfilenodeSubid == InvalidSubTransactionId)))
+ relation->rd_firstRelfilenodeSubid == InvalidSubTransactionId))))

You can remove one pair of parentheses for readability as follows:

#define RelationNeedsWAL(relation) \
+ (wal_level != WAL_LEVEL_NONE && \
(relation)->rd_rel->relpersistence == RELPERSISTENCE_PERMANENT && \
(XLogIsNeeded() || \
(relation->rd_createSubid == InvalidSubTransactionId && \
- relation->rd_firstRelfilenodeSubid == InvalidSubTransactionId)))
+ relation->rd_firstRelfilenodeSubid == InvalidSubTransactionId)))

(2)
/*
+ * Detect if the server previously crashed under wal_level='none' or not.
+ */
+ if (ControlFile->wal_level == WAL_LEVEL_NONE &&
+ (ControlFile->state != DB_SHUTDOWNED && ControlFile->state != DB_SHUTDOWNED_IN_RECOVERY))
+ ereport(ERROR,
+ (errmsg("detected an unexpected server shutdown when WAL logging was disabled"),
+ errhint("It looks like you need to deploy a new cluster from your full backup again.")));
+
+ /*
* Check that contents look valid.
*/
if (!XRecOffIsValid(ControlFile->checkPoint))

I think the above safeguard should be placed right before the following code, because it's better to first check control file contents and emit the message for the previous shutdown state.

#ifdef XLOG_REPLAY_DELAY
if (ControlFile->state != DB_SHUTDOWNED)
pg_usleep(60000000L);
#endif

(3)
@@ -449,6 +449,13 @@ XLogInsert(RmgrId rmid, uint8 info)
return EndPos;
}

+ /* Issues WAL only for XLOG resources */
+ if (wal_level == WAL_LEVEL_NONE && rmid != RM_XLOG_ID)
+ {
+ XLogResetInsertion();
+ return GetLatestCheckPointLSN();
+ }

It should be correct to use GetXLogInsertRecPtr() instead of GetLatestCheckPointLSN(), because what XLogInsert() should return is the end position of last WAL record inserted (= start position of next WAL).

Why don't we add transaction completion WAL records? That is, add "rmid != RM_XACT_ID" to the if condition. What we really want to eliminate for performance is the data modification WAL records. This might allow us to insert special checks to prevent PREPARE TRANSACTION and other stuff.

(4)
@@ -404,6 +404,10 @@ pg_logical_emit_message_bytea(PG_FUNCTION_ARGS)
bytea *data = PG_GETARG_BYTEA_PP(2);
XLogRecPtr lsn;

+ if (wal_level < WAL_LEVEL_REPLICA)
+ ereport(ERROR,
+ errmsg("propagating a message requires wal_level \"replica\" or \"logical\""));
+
lsn = LogLogicalMessage(prefix, VARDATA_ANY(data), VARSIZE_ANY_EXHDR(data),
transactional);
PG_RETURN_LSN(lsn);

@@ -192,6 +192,10 @@ replorigin_check_prerequisites(bool check_slots, bool recoveryOK)
(errcode(ERRCODE_READ_ONLY_SQL_TRANSACTION),
errmsg("cannot manipulate replication origins during recovery")));

+ if (wal_level < WAL_LEVEL_REPLICA)
+ ereport(ERROR,
+ errmsg("creating replication origins requires wal_level \"replica\" or \"logical\""));
+
}

@@ -625,6 +625,9 @@ standard_ProcessUtility(PlannedStmt *pstmt,
break;

case TRANS_STMT_PREPARE:
+ if (wal_level == WAL_LEVEL_NONE)
+ ereport(ERROR,
+ errmsg("cannot execute PREPARE TRANSACTION when WAL logging is disabled"));

What happens without these checks?

Regards
Takayuki Tsunakawa

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Masahiro Ikeda 2020-11-19 08:03:20 Re: Add statistics to pg_stat_wal view for wal related parameter tuning
Previous Message Laurenz Albe 2020-11-19 07:50:24 Re: Disable WAL logging to speed up data loading