From 79d665f1bda7944cae7631b9aeb6870fe1df67cd Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Tue, 4 Oct 2016 14:38:05 +0900 Subject: [PATCH 1/2] Rename pg_xlog to pg_wal Additional check handling is needed in pg_upgrade regarding the clusters worked on, except that it is a very mechanical patch. As pg_basebackup needs to support servers down to 9.1, special handling is needed depending on the version of the target server as a consequence of this renaming: - Soft links need to be generated using pg_xlog or pg_wal. - stream mode needs to have its output in the correct place. This has as direct consequence to need a connection to target server before generating a soft link, which is a bit earlier than before this renaming. --- doc/src/sgml/backup.sgml | 28 ++++---- doc/src/sgml/config.sgml | 6 +- doc/src/sgml/func.sgml | 2 +- doc/src/sgml/high-availability.sgml | 14 ++-- doc/src/sgml/perform.sgml | 2 +- doc/src/sgml/protocol.sgml | 6 +- doc/src/sgml/ref/pg_resetxlog.sgml | 10 +-- doc/src/sgml/ref/pg_rewind.sgml | 4 +- doc/src/sgml/ref/pg_xlogdump.sgml | 2 +- doc/src/sgml/ref/pgtestfsync.sgml | 4 +- doc/src/sgml/ref/pgupgrade.sgml | 4 +- doc/src/sgml/storage.sgml | 2 +- doc/src/sgml/wal.sgml | 10 +-- src/backend/access/transam/timeline.c | 4 +- src/backend/access/transam/xlog.c | 100 +++++++++++++-------------- src/backend/access/transam/xlogarchive.c | 2 +- src/backend/access/transam/xlogfuncs.c | 2 +- src/backend/replication/README | 2 +- src/backend/replication/basebackup.c | 20 +++--- src/backend/replication/walreceiver.c | 6 +- src/backend/replication/walsender.c | 4 +- src/backend/storage/file/fd.c | 16 ++--- src/bin/initdb/initdb.c | 12 ++-- src/bin/pg_basebackup/pg_basebackup.c | 80 +++++++++++++-------- src/bin/pg_basebackup/t/010_pg_basebackup.pl | 8 +-- src/bin/pg_resetxlog/pg_resetxlog.c | 2 +- src/bin/pg_rewind/copy_fetch.c | 4 +- src/bin/pg_rewind/filemap.c | 8 +-- src/bin/pg_rewind/parsexlog.c | 2 +- src/bin/pg_rewind/t/004_pg_xlog_symlink.pl | 10 +-- src/bin/pg_upgrade/exec.c | 79 +++++++++++++-------- src/bin/pg_xlogdump/pg_xlogdump.c | 2 +- src/common/file_utils.c | 39 +++++++---- src/include/access/xlog_internal.h | 2 +- src/include/common/file_utils.h | 3 +- src/include/postmaster/pgarch.h | 2 +- 36 files changed, 278 insertions(+), 225 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index 95d0ff3..6eaed1e 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -472,7 +472,7 @@ tar -cf backup.tar /usr/local/pgsql/data At all times, PostgreSQL maintains a - write ahead log (WAL) in the pg_xlog/ + write ahead log (WAL) in the pg_wal/ subdirectory of the cluster's data directory. The log records every change made to the database's data files. This log exists primarily for crash-safety purposes: if the system crashes, the @@ -616,7 +616,7 @@ archive_command = 'copy "%p" "C:\\server\\archivedir\\%f"' # Windows %p and %f parameters have been replaced, the actual command executed might look like this: -test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_xlog/00000001000000A900000065 /mnt/server/archivedir/00000001000000A900000065 +test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_wal/00000001000000A900000065 /mnt/server/archivedir/00000001000000A900000065 A similar command will be generated for each new file to be archived. @@ -668,9 +668,9 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_xlog/ fills, nothing further can be archived until the tape is swapped. You should ensure that any error condition or request to a human operator is reported appropriately so that the situation can be - resolved reasonably quickly. The pg_xlog/ directory will + resolved reasonably quickly. The pg_wal/ directory will continue to fill with WAL segment files until the situation is resolved. - (If the file system containing pg_xlog/ fills up, + (If the file system containing pg_wal/ fills up, PostgreSQL will do a PANIC shutdown. No committed transactions will be lost, but the database will remain offline until you free some space.) @@ -682,7 +682,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_xlog/ operation continues even if the archiving process falls a little behind. If archiving falls significantly behind, this will increase the amount of data that would be lost in the event of a disaster. It will also mean that - the pg_xlog/ directory will contain large numbers of + the pg_wal/ directory will contain large numbers of not-yet-archived segment files, which could eventually exceed available disk space. You are advised to monitor the archiving process to ensure that it is working as you intend. @@ -743,7 +743,7 @@ test ! -f /mnt/server/archivedir/00000001000000A900000065 && cp pg_xlog/ configuration file reload. If you wish to temporarily stop archiving, one way to do it is to set archive_command to the empty string (''). - This will cause WAL files to accumulate in pg_xlog/ until a + This will cause WAL files to accumulate in pg_wal/ until a working archive_command is re-established. @@ -1062,10 +1062,10 @@ SELECT pg_stop_backup(); You should, however, omit from the backup the files within the - cluster's pg_xlog/ subdirectory. This + cluster's pg_wal/ subdirectory. This slight adjustment is worthwhile because it reduces the risk of mistakes when restoring. This is easy to arrange if - pg_xlog/ is a symbolic link pointing to someplace outside + pg_wal/ is a symbolic link pointing to someplace outside the cluster directory, which is a common setup anyway for performance reasons. You might also want to exclude postmaster.pid and postmaster.opts, which record information @@ -1149,7 +1149,7 @@ SELECT pg_stop_backup(); location in case you need them later. Note that this precaution will require that you have enough free space on your system to hold two copies of your existing database. If you do not have enough space, - you should at least save the contents of the cluster's pg_xlog + you should at least save the contents of the cluster's pg_wal subdirectory, as it might contain logs which were not archived before the system went down. @@ -1172,9 +1172,9 @@ SELECT pg_stop_backup(); - Remove any files present in pg_xlog/; these came from the + Remove any files present in pg_wal/; these came from the file system backup and are therefore probably obsolete rather than current. - If you didn't archive pg_xlog/ at all, then recreate + If you didn't archive pg_wal/ at all, then recreate it with proper permissions, being careful to ensure that you re-establish it as a symbolic link if you had it set up that way before. @@ -1183,7 +1183,7 @@ SELECT pg_stop_backup(); If you have unarchived WAL segment files that you saved in step 2, - copy them into pg_xlog/. (It is best to copy them, + copy them into pg_wal/. (It is best to copy them, not move them, so you still have the unmodified files if a problem occurs and you have to start over.) @@ -1265,9 +1265,9 @@ restore_command = 'cp /mnt/server/archivedir/%f %p' WAL segments that cannot be found in the archive will be sought in - pg_xlog/; this allows use of recent un-archived segments. + pg_wal/; this allows use of recent un-archived segments. However, segments that are available from the archive will be used in - preference to files in pg_xlog/. + preference to files in pg_wal/. diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index e826c19..1bb501c 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -2932,7 +2932,7 @@ include_dir 'conf.d' Specifies the minimum number of past log file segments kept in the - pg_xlog + pg_wal directory, in case a standby server needs to fetch them for streaming replication. Each segment is normally 16 megabytes. If a standby server connected to the sending server falls behind by more than @@ -2946,7 +2946,7 @@ include_dir 'conf.d' This sets only the minimum number of segments retained in - pg_xlog; the system might need to retain more segments + pg_wal; the system might need to retain more segments for WAL archival or to recover from a checkpoint. If wal_keep_segments is zero (the default), the system doesn't keep any extra segments for standby purposes, so the number @@ -3322,7 +3322,7 @@ include_dir 'conf.d' Specify how long the standby server should wait when WAL data is not available from any sources (streaming replication, - local pg_xlog or WAL archive) before retrying to + local pg_wal or WAL archive) before retrying to retrieve WAL data. This parameter can only be set in the postgresql.conf file or on the server command line. The default value is 5 seconds. Units are milliseconds if not specified. diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index a588350..2e64cc4 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -15327,7 +15327,7 @@ SELECT * FROM pg_ls_dir('.') WITH ORDINALITY AS t(ls,n); pg_snapshots | 13 pg_multixact | 14 PG_VERSION | 15 - pg_xlog | 16 + pg_wal | 16 pg_hba.conf | 17 pg_stat_tmp | 18 pg_subtrans | 19 diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index 06f49db..5bedaf2 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -594,24 +594,24 @@ protocol to make nodes agree on a serializable transactional order. (see ) or directly from the master over a TCP connection (streaming replication). The standby server will also attempt to restore any WAL found in the standby cluster's - pg_xlog directory. That typically happens after a server + pg_wal directory. That typically happens after a server restart, when the standby replays again WAL that was streamed from the master before the restart, but you can also manually copy files to - pg_xlog at any time to have them replayed. + pg_wal at any time to have them replayed. At startup, the standby begins by restoring all WAL available in the archive location, calling restore_command. Once it reaches the end of WAL available there and restore_command - fails, it tries to restore any WAL available in the pg_xlog directory. + fails, it tries to restore any WAL available in the pg_wal directory. If that fails, and streaming replication has been configured, the standby tries to connect to the primary server and start streaming WAL - from the last valid record found in archive or pg_xlog. If that fails + from the last valid record found in archive or pg_wal. If that fails or streaming replication is not configured, or if the connection is later disconnected, the standby goes back to step 1 and tries to restore the file from the archive again. This loop of retries from the - archive, pg_xlog, and via streaming replication goes on until the server + archive, pg_wal, and via streaming replication goes on until the server is stopped or failover is triggered by a trigger file. @@ -619,7 +619,7 @@ protocol to make nodes agree on a serializable transactional order. Standby mode is exited and the server switches to normal operation when pg_ctl promote is run or a trigger file is found (trigger_file). Before failover, - any WAL immediately available in the archive or in pg_xlog will be + any WAL immediately available in the archive or in pg_wal will be restored, but no attempt is made to connect to the master. @@ -895,7 +895,7 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass' However, these methods often result in retaining more WAL segments than required, whereas replication slots retain only the number of segments known to be needed. An advantage of these methods is that they bound - the space requirement for pg_xlog; there is currently no way + the space requirement for pg_wal; there is currently no way to do this using replication slots. diff --git a/doc/src/sgml/perform.sgml b/doc/src/sgml/perform.sgml index 7bcbfa7..8d30fd1 100644 --- a/doc/src/sgml/perform.sgml +++ b/doc/src/sgml/perform.sgml @@ -1612,7 +1612,7 @@ SELECT * FROM x, y, a, b, c WHERE something AND somethingelse; Increase and ; this reduces the frequency of checkpoints, but increases the storage requirements of - /pg_xlog. + /pg_wal. diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index 3384e73..50cf527 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -1947,7 +1947,7 @@ The commands accepted in walsender mode are: Include the necessary WAL segments in the backup. This will include all the files between start and stop backup in the - pg_xlog directory of the base directory tar + pg_wal directory of the base directory tar file. @@ -2076,8 +2076,8 @@ The commands accepted in walsender mode are: - pg_xlog, including subdirectories. If the backup is run - with WAL files included, a synthesized version of pg_xlog will be + pg_wal, including subdirectories. If the backup is run + with WAL files included, a synthesized version of pg_wal will be included, but it will only contain the files necessary for the backup to work, not the rest of the contents. diff --git a/doc/src/sgml/ref/pg_resetxlog.sgml b/doc/src/sgml/ref/pg_resetxlog.sgml index fd9d0be..c949c5e 100644 --- a/doc/src/sgml/ref/pg_resetxlog.sgml +++ b/doc/src/sgml/ref/pg_resetxlog.sgml @@ -173,22 +173,22 @@ PostgreSQL documentation The WAL starting address should be larger than any WAL segment file name currently existing in - the directory pg_xlog under the data directory. + the directory pg_wal under the data directory. These names are also in hexadecimal and have three parts. The first part is the timeline ID and should usually be kept the same. For example, if 00000001000000320000004A is the - largest entry in pg_xlog, use -l 00000001000000320000004B or higher. + largest entry in pg_wal, use -l 00000001000000320000004B or higher. pg_resetxlog itself looks at the files in - pg_xlog and chooses a default diff --git a/doc/src/sgml/ref/pg_rewind.sgml b/doc/src/sgml/ref/pg_rewind.sgml index 42ebfbf..371c4a4 100644 --- a/doc/src/sgml/ref/pg_rewind.sgml +++ b/doc/src/sgml/ref/pg_rewind.sgml @@ -61,14 +61,14 @@ PostgreSQL documentation pg_rewind examines the timeline histories of the source and target clusters to determine the point where they diverged, and - expects to find WAL in the target cluster's pg_xlog directory + expects to find WAL in the target cluster's pg_wal directory reaching all the way back to the point of divergence. The point of divergence can be found either on the target timeline, the source timeline, or their common ancestor. In the typical failover scenario where the target cluster was shut down soon after the divergence, this is not a problem, but if the target cluster ran for a long time after the divergence, the old WAL files might no longer be present. In that case, they can be manually - copied from the WAL archive to the pg_xlog directory, or + copied from the WAL archive to the pg_wal directory, or fetched on startup by configuring recovery.conf. The use of pg_rewind is not limited to failover, e.g. a standby server can be promoted, run some write transactions, and then rewinded diff --git a/doc/src/sgml/ref/pg_xlogdump.sgml b/doc/src/sgml/ref/pg_xlogdump.sgml index 177caab..cfb6d87 100644 --- a/doc/src/sgml/ref/pg_xlogdump.sgml +++ b/doc/src/sgml/ref/pg_xlogdump.sgml @@ -118,7 +118,7 @@ PostgreSQL documentation Directory in which to find log segment files. The default is to search - for them in the pg_xlog subdirectory of the current + for them in the pg_wal subdirectory of the current directory. diff --git a/doc/src/sgml/ref/pgtestfsync.sgml b/doc/src/sgml/ref/pgtestfsync.sgml index 6e134c7..5856356 100644 --- a/doc/src/sgml/ref/pgtestfsync.sgml +++ b/doc/src/sgml/ref/pgtestfsync.sgml @@ -57,8 +57,8 @@ Specifies the file name to write test data in. This file should be in the same file system that the - pg_xlog directory is or will be placed in. - (pg_xlog contains the WAL files.) + pg_wal directory is or will be placed in. + (pg_wal contains the WAL files.) The default is pg_test_fsync.out in the current directory. diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index 9685193..4b9aab2 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -345,7 +345,7 @@ NET STOP postgresql-9.0 your old cluster once you start the new cluster after the upgrade. Link mode also requires that the old and new cluster data directories be in the - same file system. (Tablespaces and pg_xlog can be on + same file system. (Tablespaces and pg_wal can be on different file systems.) See pg_upgrade --help for a full list of options. @@ -508,7 +508,7 @@ rsync --archive --delete --hard-links --size-only old_pgdata new_pgdata remote_d If you have tablespaces, you will need to run a similar rsync command for each tablespace directory. If you - have relocated pg_xlog outside the data directories, + have relocated pg_wal outside the data directories, rsync must be run on those directories too. diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 1b812bd..fddb69b 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -141,7 +141,7 @@ Item - pg_xlog + pg_wal Subdirectory containing WAL (Write Ahead Log) files diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml index fe3b588..346aa76 100644 --- a/doc/src/sgml/wal.sgml +++ b/doc/src/sgml/wal.sgml @@ -557,7 +557,7 @@ - The number of WAL segment files in pg_xlog directory depends on + The number of WAL segment files in pg_wal directory depends on min_wal_size, max_wal_size and the amount of WAL generated in previous checkpoint cycles. When old log segment files are no longer needed, they are removed or recycled (that is, @@ -582,7 +582,7 @@ kept at all times. Also, if WAL archiving is used, old segments can not be removed or recycled until they are archived. If WAL archiving cannot keep up with the pace that WAL is generated, or if archive_command - fails repeatedly, old WAL files will accumulate in pg_xlog + fails repeatedly, old WAL files will accumulate in pg_wal until the situation is resolved. A slow or failed standby server that uses a replication slot will have the same effect (see ). @@ -594,7 +594,7 @@ which are similar to checkpoints in normal operation: the server forces all its state to disk, updates the pg_control file to indicate that the already-processed WAL data need not be scanned again, - and then recycles any old log segment files in the pg_xlog + and then recycles any old log segment files in the pg_wal directory. Restartpoints can't be performed more frequently than checkpoints in the master because restartpoints can only be performed at checkpoint records. @@ -750,7 +750,7 @@ WAL logs are stored in the directory - pg_xlog under the data directory, as a set of + pg_wal under the data directory, as a set of segment files, normally each 16 MB in size (but the size can be changed by altering the