From 51512cd9cb59d169b041d10d62fc6a282011675c Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Wed, 14 Sep 2022 21:26:10 +0200 Subject: [PATCH] Restore archive_command documentation Commit 5ef1eefd76f404ddc59b885d50340e602b70f05f, which added archive_library, purged most mentions of archive_command from the documentation. This is inappropriate, since archive_command is still a feature in use and users will want to see information about it. This restores all the removed mentions and rephrases things so that archive_command and archive_library are presented as alternatives of each other. --- doc/src/sgml/backup.sgml | 50 +++++++++++++++++-------- doc/src/sgml/config.sgml | 58 +++++++++++++++-------------- doc/src/sgml/high-availability.sgml | 6 +-- doc/src/sgml/ref/pg_receivewal.sgml | 7 +++- doc/src/sgml/wal.sgml | 3 +- 5 files changed, 76 insertions(+), 48 deletions(-) diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index dee59bb422..a6d7105836 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -593,7 +593,7 @@ Setting Up WAL Archiving provide the database administrator with flexibility, PostgreSQL tries not to make any assumptions about how the archiving will be done. Instead, PostgreSQL lets - the administrator specify an archive library to be executed to copy a + the administrator specify a shell command or an archive library to be executed to copy a completed segment file to wherever it needs to go. This could be as simple as a shell command that uses cp, or it could invoke a complex C function — it's all up to you. @@ -603,13 +603,15 @@ Setting Up WAL Archiving To enable WAL archiving, set the configuration parameter to replica or higher, to on, - and specify the library to use in the configuration parameter + or specify the library to use in the configuration parameter. In practice these settings will always be placed in the postgresql.conf file. - One simple way to archive is to set archive_library to - an empty string and to specify a shell command in - . + + + In archive_command, %p is replaced by the path name of the file to archive, while %f is replaced by only the file name. @@ -633,6 +635,24 @@ Setting Up WAL Archiving A similar command will be generated for each new file to be archived. + + It is important that the archive command return zero exit status if and + only if it succeeds. Upon getting a zero result, + PostgreSQL will assume that the file has been + successfully archived, and will remove or recycle it. However, a nonzero + status tells PostgreSQL that the file was not archived; + it will try again periodically until it succeeds. + + + + When the archive command is terminated by a signal (other than + SIGTERM that is used as part of a server + shutdown) or an error by the shell with an exit status greater than + 125 (such as command not found), the archiver process aborts and gets + restarted by the postmaster. In such cases, the failure is + not reported in . + + Another way to archive is to use a custom archive module as the archive_library. Since such modules are written in @@ -678,7 +698,7 @@ Setting Up WAL Archiving - The archive library should generally be designed to refuse to overwrite + Archive commands and libraries should generally be designed to refuse to overwrite any pre-existing archive file. This is an important safety feature to preserve the integrity of your archive in case of administrator error (such as sending the output of two different servers to the same archive @@ -686,9 +706,9 @@ Setting Up WAL Archiving - It is advisable to test your proposed archive library to ensure that it + It is advisable to test your proposed archive command or library to ensure that it indeed does not overwrite an existing file, and that it returns - false in this case. + nonzero status or false, respectively, in this case. The example command above for Unix ensures this by including a separate test step. On some Unix platforms, cp has switches such as that can be used to do the same thing @@ -700,7 +720,7 @@ Setting Up WAL Archiving While designing your archiving setup, consider what will happen if - the archive library fails repeatedly because some aspect requires + the archive command or library fails repeatedly because some aspect requires operator intervention or the archive runs out of space. For example, this could occur if you write to tape without an autochanger; when the tape fills, nothing further can be archived until the tape is swapped. @@ -715,7 +735,7 @@ Setting Up WAL Archiving - The speed of the archive library is unimportant as long as it can keep up + The speed of the archive command or library is unimportant as long as it can keep up with the average rate at which your server generates WAL data. Normal operation continues even if the archiving process falls a little behind. If archiving falls significantly behind, this will increase the amount of @@ -727,7 +747,7 @@ Setting Up WAL Archiving - In writing your archive library, you should assume that the file names to + In writing your archive command or library, you should assume that the file names to be archived can be up to 64 characters long and can contain any combination of ASCII letters, digits, and dots. It is not necessary to preserve the original relative path but it is necessary to preserve the file @@ -748,7 +768,7 @@ Setting Up WAL Archiving - The archive function is only invoked on completed WAL segments. Hence, + The archive command or function is only invoked on completed WAL segments. Hence, if your server generates only little WAL traffic (or has slack periods where it does so), there could be a long delay between the completion of a transaction and its safe recording in archive storage. To put @@ -777,7 +797,7 @@ Setting Up WAL Archiving turned on during execution of one of these statements, WAL would not contain enough information for archive recovery. (Crash recovery is unaffected.) For this reason, wal_level can only be changed at - server start. However, archive_library can be changed with a + server start. However, archive_command and archive_library can be changed with a configuration file reload. If you are archiving via shell and wish to temporarily stop archiving, one way to do it is to set archive_command to the empty @@ -947,12 +967,12 @@ Making a Base Backup Using the Low Level API On a standby, archive_mode must be always in order for pg_backup_stop to wait. Archiving of these files happens automatically since you have - already configured archive_library or + already configured archive_command or archive_library or archive_command. In most cases this happens quickly, but you are advised to monitor your archive system to ensure there are no delays. If the archive process has fallen behind because of failures of the - archive library or archive command, it will keep retrying + archive command or library, it will keep retrying until the archive succeeds and the backup is complete. If you wish to place a time limit on the execution of pg_backup_stop, set an appropriate diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 5aac7110b1..09c7a6116a 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -3496,7 +3496,7 @@ Checkpoints Maximum size to let the WAL grow during automatic checkpoints. This is a soft limit; WAL size can exceed max_wal_size under special circumstances, such as - heavy load, a failing archive_library, or a high + heavy load, a failing archive_command or archive_library, or a high wal_keep_size setting. If this value is specified without units, it is taken as megabytes. The default is 1 GB. @@ -3545,6 +3545,7 @@ Archiving When archive_mode is enabled, completed WAL segments are sent to archive storage by setting + or . In addition to off, to disable, there are two modes: on, and always. During normal operation, there is no @@ -3555,6 +3556,9 @@ Archiving for details. + archive_mode and archive_command are + separate variables so that archive_command can be + changed without leaving archiving mode. This parameter can only be set at server start. archive_mode cannot be enabled when wal_level is set to minimal. @@ -3562,28 +3566,6 @@ Archiving - - archive_library (string) - - archive_library configuration parameter - - - - - The library to use for archiving completed WAL file segments. If set to - an empty string (the default), archiving via shell is enabled, and - is used. Otherwise, the specified - shared library is used for archiving. For more information, see - and - . - - - This parameter can only be set in the - postgresql.conf file or on the server command line. - - - - archive_command (string) @@ -3607,10 +3589,10 @@ Archiving This parameter can only be set in the postgresql.conf file or on the server command line. It is ignored unless archive_mode was enabled at server start and - archive_library specifies to archive via shell command. + archive_library is set to an empty string. If archive_command is an empty string (the default) while - archive_mode is enabled and archive_library - specifies archiving via shell, WAL archiving is temporarily + archive_mode is enabled (and archive_library + is set to an empty string), WAL archiving is temporarily disabled, but the server continues to accumulate WAL segment files in the expectation that a command will soon be provided. Setting archive_command to a command that does nothing but @@ -3622,6 +3604,28 @@ Archiving + + archive_library (string) + + archive_library configuration parameter + + + + + The library to use for archiving completed WAL file segments. If set to + an empty string (the default), archiving via shell is enabled, and + is used. Otherwise, the specified + shared library is used for archiving. For more information, see + and + . + + + This parameter can only be set in the + postgresql.conf file or on the server command line. + + + + archive_timeout (integer) @@ -3630,7 +3634,7 @@ Archiving - The is only invoked for + The or is only invoked for completed WAL segments. Hence, if your server generates little WAL traffic (or has slack periods where it does so), there could be a long delay between the completion of a transaction and its safe diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index 3df4cda716..b2b3129397 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -935,7 +935,7 @@ Replication Slots In lieu of using replication slots, it is possible to prevent the removal of old WAL segments using , or by storing the segments in an archive using - . + or . However, these methods often result in retaining more WAL segments than required, whereas replication slots retain only the number of segments known to be needed. On the other hand, replication slots can retain so @@ -1386,10 +1386,10 @@ Continuous Archiving in Standby to always, and the standby will call the archive command for every WAL segment it receives, whether it's by restoring from the archive or by streaming replication. The shared archive can - be handled similarly, but the archive_library must + be handled similarly, but the archive_command or archive_library must test if the file being archived exists already, and if the existing file has identical contents. This requires more care in the - archive_library, as it must + archive_command or archive_library, as it must be careful to not overwrite an existing file with different contents, but return success if the exactly same file is archived twice. And all that must be done free of race conditions, if two servers attempt diff --git a/doc/src/sgml/ref/pg_receivewal.sgml b/doc/src/sgml/ref/pg_receivewal.sgml index 4fe9e1a874..7138a8e3f6 100644 --- a/doc/src/sgml/ref/pg_receivewal.sgml +++ b/doc/src/sgml/ref/pg_receivewal.sgml @@ -40,7 +40,8 @@ Description pg_receivewal streams the write-ahead log in real time as it's being generated on the server, and does not wait - for segments to complete like does. + for segments to complete like and + does. For this reason, it is not necessary to set when using pg_receivewal. @@ -486,11 +487,13 @@ Notes When using pg_receivewal instead of + or as the main WAL backup method, it is strongly recommended to use replication slots. Otherwise, the server is free to recycle or remove write-ahead log files before they are backed up, because it does not have any information, either - from or the replication slots, about + from or + or the replication slots, about how far the WAL stream has been archived. Note, however, that a replication slot will fill up the server's disk space if the receiver does not keep up with fetching the WAL data. diff --git a/doc/src/sgml/wal.sgml b/doc/src/sgml/wal.sgml index 4b6ef283c1..27fb020a06 100644 --- a/doc/src/sgml/wal.sgml +++ b/doc/src/sgml/wal.sgml @@ -636,7 +636,8 @@ <acronym>WAL</acronym> Configuration WAL files plus one additional WAL file are kept at all times. Also, if WAL archiving is used, old segments cannot be removed or recycled until they are archived. If WAL archiving cannot keep up - with the pace that WAL is generated, or if archive_library + with the pace that WAL is generated, or if archive_command + or archive_library fails repeatedly, old WAL files will accumulate in pg_wal until the situation is resolved. A slow or failed standby server that uses a replication slot will have the same effect (see -- 2.37.3