== PostgreSQL Weekly News - October 6, 2019 ==

== PostgreSQL Weekly News - October 6, 2019 ==

PostgreSQL 12 released!
https://www.postgresql.org/about/news/1976/

== PostgreSQL Product News ==

pgBackRest 2.18, a backup and restore system for PostgreSQL, released.
https://pgbackrest.org/release.html#2.18

PGDay Down Under 2019 will be held on December 15, 2019 in Sydney, Australia.
Registration is open, and Early Bird tickets are available through October 31.
https://pgdu.org/

== PostgreSQL Jobs for October ==

http://archives.postgresql.org/pgsql-jobs/2019-10/

== PostgreSQL Local ==

PostgresConf South Africa 2019 will take place in Johannesburg on October 8-9, 2019
https://postgresconf.org/conferences/SouthAfrica2019

PostgreSQL Conference Europe 2019 will be held on October 15-18, 2019 in Milan,
Italy.
https://2019.pgconf.eu/

2Q PGConf 2019 will be held December 4-5 in Chicago.
https://www.2qpgconf.com/

PGDay Down Under 2019 will be held on November 15, 2019 in Sydney,
Australia.
https://pgdu.org/

pgDay Paris 2020 will be held in Paris, France on March 26, 2020
at Espace Saint-Martin.
http://2020.pgday.paris/

Nordic PGDay 2020 will be held in Helsinki, Finland at the Hilton Helsinki
Strand Hotel on March 24, 2020. The CfP is open through December 31, 2019 at
https://2020.nordicpgday.org/cfp/

PGConf India 2020 will be on February 26-28, 2020 in Bengaluru, Karnataka. The
CfP is open until November 15, 2019.
http://pgconf.in/

The German-speaking PostgreSQL Conference 2020 will take place on May 10, 2019
in Stuttgart.

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Peter Eisentraut pushed:

- doc: Add a link target. Forward-patched from PostgreSQL 12 release notes
patch, for consistency.
https://git.postgresql.org/pg/commitdiff/92f1545d6ea9fbfe4b47108028ccaae351a1480c

- doc: Further clarify how recovery target parameters are applied. Recovery
target parameters are all applied even in standby mode. The previous
documentation mostly wished they were not but this was never the case.
Discussion:
https://www.postgresql.org/message-id/flat/e445616d-023e-a268-8aa1-67b8b335340c%40pgmasters.net
https://git.postgresql.org/pg/commitdiff/e04a53a6071d13ef4a13a41c6419d8e14c30b95c

Tom Lane pushed:

- Fix bogus order of error checks in new channel_binding code. Coverity pointed
out that it's pretty silly to check for a null pointer after we've already
dereferenced the pointer. To fix, just swap the order of the two error
checks. Oversight in commit d6e612f83.
https://git.postgresql.org/pg/commitdiff/2c97f73468419672f2340afb24f1321695ee3002

- Doc: improve PREPARE documentation, cross-referencing to plan_cache_mode. The
behavior described in the PREPARE man page applies only for the default
plan_cache_mode setting, so explain that properly. Rewrite some of the text
while I'm here. Per suggestion from Bruce. Discussion:
https://postgr.es/m/20190930155505.GA21095@momjian.us
https://git.postgresql.org/pg/commitdiff/ce734aaec1891ca2180c269b4c9adbfb13ca4052

- Rely on plan_cache_mode to force generic plans in partition_prune test. This
file had a very weird mix of tests that did "set plan_cache_mode =
force_generic_plan" to get a generic plan, and tests that relied on using five
dummy executions of a prepared statement. Converting them all to rely on
plan_cache_mode is more consistent and shaves off a noticeable fraction of the
test script's runtime. Discussion:
https://postgr.es/m/11952.1569536725@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/d52eaa094847d395f942827a6f413904e516994c

- Allow repalloc() to give back space when a large chunk is downsized. Up to
now, if you resized a large (>8K) palloc chunk down to a smaller size, aset.c
made no attempt to return any space to the malloc pool. That's unpleasant if a
really large allocation is resized to a significantly smaller size. I think
no such cases existed when this code was designed, and I'm not sure whether
they're common even yet, but an upcoming fix to encoding conversion will
certainly create such cases. Therefore, fix AllocSetRealloc so that it gives
realloc() a chance to do something with the block. This doesn't noticeably
increase complexity, we mostly just have to change the order in which the
cases are considered. Back-patch to all supported branches. Discussion:
https://postgr.es/m/20190816181418.GA898@alvherre.pgsql Discussion:
https://postgr.es/m/3614.1569359690@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/c477f3e449d1a7322c2453e9cd9d6b710ae91577

- Avoid unnecessary out-of-memory errors during encoding conversion. Encoding
conversion uses the very simplistic rule that the output can't be more than 4X
longer than the input, and palloc's a buffer of that size. This results in
failure to convert any string longer than 1/4 GB, which is becoming an
annoying limitation. As a band-aid to improve matters, allow the allocated
output buffer size to exceed 1GB. We still insist that the final result fit
into MaxAllocSize (1GB), though. Perhaps it'd be safe to relax that
restriction, but it'd require close analysis of all callers, which is daunting
(not least because external modules might call these functions). For the
moment, this should allow a 2X to 4X improvement in the longest string we can
convert, which is a useful gain in return for quite a simple patch. Also,
once we have successfully converted a long string, repalloc the output down to
the actual string length, returning the excess to the malloc pool. This seems
worth doing since we can usually expect to give back several MB if we take
this path at all. This still leaves much to be desired, most notably that the
assumption that MAX_CONVERSION_GROWTH == 4 is very fragile, and yet we have no
guard code verifying that the output buffer isn't overrun. Fixing that would
require significant changes in the encoding conversion APIs, so it'll have to
wait for some other day. The present patch seems safely back-patchable, so
patch all supported branches. Alvaro Herrera and Tom Lane Discussion:
https://postgr.es/m/20190816181418.GA898@alvherre.pgsql Discussion:
https://postgr.es/m/3614.1569359690@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/8e10405c745003c5c16acb2da847db9bed1a169e

- Fix bitshiftright()'s zero-padding some more. Commit 5ac0d9360 failed to
entirely fix bitshiftright's habit of leaving one-bits in the pad space that
should be all zeroes, because in a moment of sheer brain fade I'd concluded
that only the code path used for not-a-multiple-of-8 shift distances needed to
be fixed. Of course, a multiple-of-8 shift distance can also cause the
problem, so we need to forcibly zero the extra bits in both cases. Per bug
#16037 from Alexander Lakhin. As before, back-patch to all supported
branches. Discussion:
https://postgr.es/m/16037-1d1ebca564db54f4@postgresql.org
https://git.postgresql.org/pg/commitdiff/61aa9f544a91f2908e4c7cd549907cdc5b6f1c82

- Avoid use of wildcard in pg_waldump's .gitignore. This would be all right,
maybe, if it didn't also match a file that definitely should not be ignored.
We don't add rmgrs so often that manual maintenance of this file list is
impractical, so just write out the list. (I find the equivalent wildcard use
in the Makefile pretty lazy and unsafe as well, but will leave that alone
until it actually causes a problem.) Per bug #16042 from Denis Stuchalin.
Discussion: https://postgr.es/m/16042-c174ee692ac21cbd@postgresql.org
https://git.postgresql.org/pg/commitdiff/d82f3909da11f9732fbc488333de0fdeb4d91ff5

- Avoid trying to release a List's initial allocation via repalloc(). Commit
1cff1b95a included some code that supposed it could repalloc() a memory chunk
to a smaller size without risk of the chunk moving. That was not a great idea,
because it depended on undocumented behavior of AllocSetRealloc, which commit
c477f3e44 changed thereby breaking it. (Not to mention that this code ought to
work with other memory context types, which might not work the same...) So
get rid of the repalloc calls, and instead just wipe the now-unused ListCell
array and/or tell Valgrind it's NOACCESS, as if we'd freed it. In cases where
the initial list allocation had been quite large, this could represent an
annoying waste of space. In principle we could ameliorate that by allocating
the initial cell array separately when it exceeds some threshold. But that
would complicate new_list() which is hot code, and the returns would
materialize only in narrow cases. On balance I don't think it'd be worth it.
Discussion: https://postgr.es/m/17059.1570208426@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/ac12ab06a96179d44046494bc76ec53f30b5d30a

Andrew Dunstan pushed:

- Allow SSL TAP tests to run on Windows. Windows does not enforce key file
permissions checks in libpq, and psql can produce CRLF line endings on
Windows. Backpatch to Release 12 (CRLF) and Release 11 (permissions check)
https://git.postgresql.org/pg/commitdiff/258bf86a9aec05b531c206a6e661ec8c0754e10f

- Suppress another CR in program output. This one was exposed by a12c75a10.
Backpatch to release 11 where check_pg_config was introduced.
https://git.postgresql.org/pg/commitdiff/863fa43e32b02963f240864245c6c922f619459f

- Handle spaces in OpenSSL install location for MSVC. First, make sure that the
.exe name is quoted when trying to get the version number. Also, don't quote
the lib name for using in the project files if it's already been quoted. This
second change applies to all libraries, not just OpenSSL. This has clearly
been broken forever, so backpatch to all live branches.
https://git.postgresql.org/pg/commitdiff/ad7595b890dbc26284bb0d784c2aaf1b9d6f903a

Andres Freund pushed:

- Fix determination when slot types for upper executor nodes are fixed. For many
queries the fact that the tuple descriptor from the lower node was not taken
into account when determining whether the type of a slot is fixed, lead to
tuple deforming for such upper nodes not to be JIT accelerated. I broke this
in 675af5c01e297. There is ongoing work to enable writing regression tests
for related behavior (including a patch that would have detected this
regression), by optionally showing such details in EXPLAIN. But as it seems
unlikely that that will be suitable for stable branches, just merge the fix
for now. While it's fairly close to the 12 release window, the fact that 11
continues to perform JITed tuple deforming in these cases, that there's still
cases where we do so in 12, and the fact that the performance regression can
be sizable, weigh in favor of fixing it now. Author: Andres Freund
Discussion:
https://postgr.es/m/20190927072053.njf6prdl3vb7y7qb@alap3.anarazel.de
Backpatch: 12-, where 675af5c01e297 was merged.
https://git.postgresql.org/pg/commitdiff/97e971ee05d5a0f6361ea34abf27059d762045a7

- jit: Re-allow JIT compilation of execGrouping.c hashtable comparisons. In the
course of 5567d12ce03, 356687bd8 and 317ffdfeaac, I changed
BuildTupleHashTable[Ext]'s call to ExecBuildGroupingEqual to not pass in the
parent node, but NULL. Which in turn prevents the tuple equality comparator
from being JIT compiled. While that fixes bug #15486, it is not actually
necessary after all of the above commits, as we don't re-build the comparator
when using the new BuildTupleHashTableExt() interface (as the content of the
hashtable are reset, but the TupleHashTable itself is not). Therefore
re-allow jit compilation for callers that use BuildTupleHashTableExt with a
separate context for "metadata" and content. As in the previous commit,
there's ongoing work to make this easier to test to prevent such regressions
in the future, but that infrastructure is not going to be backpatchable. The
performance impact of not JIT compiling hashtable equality comparators can be
substantial e.g. for aggregation queries that aggregate a lot of input rows to
few output rows (when there are a lot of output groups, there will be fewer
comparisons). Author: Andres Freund Discussion:
https://postgr.es/m/20190927072053.njf6prdl3vb7y7qb@alap3.anarazel.de
Backpatch: 11, just as 5567d12ce03
https://git.postgresql.org/pg/commitdiff/ac88807f9b227ddcd92b8be9a053094837c1b99a

- Fix pg_rewind link order issue introduced in 927474ce1a2. The aforementioned
commit orders the link to pgfeutils after libpq, which can fail because
pgfeutils uses symbols from libpq. Per buildfarm animal jacana. Author:
Andres Freund Discussion:
https://postgr.es/m/20190930192013.r3wievljua2n3tbb@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/5daf682cfc974bf9095be527603c6410921892a9

- Reduce code duplication for ExecJust*Var operations. This is mainly in
preparation for adding further fastpath evaluation routines. Also reorder
ExecJust*Var functions to be consistent with the order in which they're used.
Author: Andres Freund Discussion:
https://postgr.es/m/CAE-ML+9OKSN71+mHtfMD-L24oDp8dGTfaVjDU6U+j+FNAW5kRQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/34c9c53bb035ba92491006eb80f7a60527db367d

- Don't generate EEOP_*_FETCHSOME operations for slots know to be virtual. That
avoids unnecessary work during both interpreted execution, and JIT compiled
expression evaluation. Both benefit from fewer expression steps needing be
processed, and for interpreted execution there now is a fastpath dedicated to
just fetching a value from a virtual slot. That's e.g. beneficial for
hashjoins over nodes that perform projections, as the hashed columns are
currently fetched individually. Author: Soumyadeep Chakraborty, Andres Freund
Discussion:
https://postgr.es/m/CAE-ML+9OKSN71+mHtfMD-L24oDp8dGTfaVjDU6U+j+FNAW5kRQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/36d22dd95bc87ca68e742da91f47f8826f8758c9

- Use a fd opened for read/write when syncing slots during startup, take 2.
Cribbing from dfbaed45975: Some operating systems, including the
reporter's windows, return EBADFD or similar when fsync() is invoked on a
O_RDONLY file descriptor. Unfortunately RestoreSlotFromDisk() does exactly
that; which causes failures after restarts in at least some scenarios.
If you hit the bug the error message will be something like ERROR: could
not fsync file "pg_replslot/$name/state": Bad file descriptor Simply use
O_RDWR instead of O_RDONLY when opening the relevant file descriptor to
fix the bug. Unfortunately this fix was undone in 82a5649fb9db. Re-apply, and
add a comment. Bug: 16039 Reported-By: Hans Buschmann Author: Andres Freund
Discussion: https://postgr.es/m/16039-196fc97cc05e141c@postgresql.org
Backpatch: 12-, as 82a5649fb9db
https://git.postgresql.org/pg/commitdiff/a586cc4b6c568e88a459f1a69ac82aa42af7e5ba

- Fix crash caused by EPQ happening with a before update trigger present. When
ExecBRUpdateTriggers()'s GetTupleForTrigger() follows an EPQ chain the former
needs to run the result tuple through the junkfilter again, and update the
slot containing the new version of the tuple to contain that new version. The
input tuple may already be in the junkfilter's output slot, which used to be
OK - we don't need the previous version anymore. Unfortunately ff11e7f4b9ae
started to use ExecCopySlot() to update newslot, and ExecCopySlot() doesn't
support copying a slot into itself, leading to a slot in a corrupt state,
which then can cause crashes or other symptoms. Fix this by skipping the
ExecCopySlot() when copying into itself. While we could have easily made
ExecCopySlot() handle that case, it seems better to add an assert forbidding
doing so instead. As the goal of copying might be to make the contents of one
slot independent from another, it seems failure prone to handle doing so
silently. A follow-up commit will add tests for the obviously under-covered
combination of EPQ and triggers. Done as a separate commit as it might make
sense to backpatch them further than this bug. Also remove confusion with
confusing variable names for slots in ExecBRDeleteTriggers() and
ExecBRUpdateTriggers(). Bug: #16036 Reported-By: Антон Власов Author: Andres
Freund Discussion: https://postgr.es/m/16036-28184c90d952fb7f@postgresql.org
Backpatch: 12-, where ff11e7f4b9ae was merged
https://git.postgresql.org/pg/commitdiff/d986d4e87f61c68f52c68ebc274960dc664b7b4e

- Add isolation tests for the combination of EPQ and triggers. As evidenced by
bug #16036 this area is woefully under-tested. Add fairly extensive tests for
the combination. Backpatch back to 9.6 - before that isolationtester was not
capable enough. While we don't backpatch tests all the time, future fixes to
trigger.c would potentially look different enough in 12+ from the earlier
branches that introducing bugs during backpatching is more likely than normal.
Also, it's just a crucial and undertested area of the code. Author: Andres
Freund Discussion: https://postgr.es/m/16036-28184c90d952fb7f@postgresql.org
Backpatch: 9.6-, the earliest these tests work
https://git.postgresql.org/pg/commitdiff/c88411995098800e19e8507d4db19e86b09d73e4

- Disable one set of tests from c8841199509. One of the upsert related tests is
unstable (sometimes even hanging until isolationtester's step timeout is
reached). Based on preliminary analysis that might be a problem outside of
just that test, but not really related to EPQ and triggers. Disable for now,
to get the buildfarm greener again. Discussion:
https://postgr.es/m/20191004222437.45qmglpto43pd3jb@alap3.anarazel.de
Backpatch: 9.6-, just like c8841199509.
https://git.postgresql.org/pg/commitdiff/6e61d75f5258dc395c131ad5edd712852e29939e

- Disable one more set of tests from c8841199509. Discussion:
https://postgr.es/m/20191004222437.45qmglpto43pd3jb@alap3.anarazel.de
Backpatch: 9.6-, just like c8841199509 and 6e61d75f525
https://git.postgresql.org/pg/commitdiff/3a68105154c3a35e4b107b41e2f54ec85fbe29f5

Fujii Masao pushed:

- Make crash recovery ignore recovery target settings. In v11 or before,
recovery target settings could not take effect in crash recovery because they
are specified in recovery.conf and crash recovery always starts without
recovery.conf. But commit 2dedf4d9a8 integrated recovery.conf into
postgresql.conf and which unexpectedly allowed recovery target settings to
take effect even in crash recovery. This is definitely not good behavior. To
fix the issue, this commit makes crash recovery always ignore recovery target
settings. Back-patch to v12. Author: Peter Eisentraut Reviewed-by: Fujii
Masao Discussion:
https://postgr.es/m/e445616d-023e-a268-8aa1-67b8b335340c@pgmasters.net
https://git.postgresql.org/pg/commitdiff/7acf8a876b7704ae162fc4f48ff97f4290fb0a61

Michaël Paquier pushed:

- Fix SSL test for libpq connection parameter channel_binding. When compiling
Postgres with OpenSSL 1.0.1 or older versions, SCRAM's channel binding cannot
be supported as X509_get_signature_nid() is needed, which causes a regression
test with channel_binding='require' to fail as the server cannot publish
SCRAM-SHA-256-PLUS as SASL mechanism over an SSL connection. Fix the issue by
using a method similar to c3d41cc, making the test result conditional. The
test passes if X509_get_signature_nid() is present, and when missing we test
for a connection failure. Testing a connection failure is more useful than
skipping the test as we should fail the connection if channel binding is
required by the client but the server does not support it. Reported-by: Tom
Lane, Michael Paquier Author: Michael Paquier Discussion:
https://postgr.es/m/20190927024457.GA8485@paquier.xyz Discussion:
https://postgr.es/m/24857.1569775891@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/a12c75a1048295f03cf85533d6dcab5072ba262b

- Fix confusing error caused by connection parameter channel_binding. When using
a client compiled without channel binding support (linking to OpenSSL 1.0.1 or
older) to connect to a server which supports channel binding (linking to
OpenSSL 1.0.2 or newer), libpq would generate a confusing error message with
channel_binding=require for an SSL connection, where the server sends back
SCRAM-SHA-256-PLUS: "channel binding is required, but server did not offer an
authentication method that supports channel binding." This is confusing
because the server did send a SASL mechanism able to support channel binding,
but libpq was not able to detect that properly. The situation can be
summarized as followed for the case described in the previous paragraph for
the SASL mechanisms used with the various modes of channel_binding: 1) Client
supports channel binding. 1-1) channel_binding = disable => OK, with
SCRAM-SHA-256. 1-2) channel_binding = prefer => OK, with SCRAM-SHA-256-PLUS.
1-3) channel_binding = require => OK, with SCRAM-SHA-256-PLUS. 2) Client does
not support channel binding. 2-1) channel_binding = disable => OK, with
SCRAM-SHA-256. 2-2) channel_binding = prefer => OK, with SCRAM-SHA-256. 2-3)
channel_binding = require => failure with new error message, instead of the
confusing one. This commit updates case 2-3 to generate a better error
message. Note that the SSL TAP tests are not impacted as it is not possible
to test with mixed versions of OpenSSL for the backend and libpq.
Reported-by: Tom Lane Author: Michael Paquier Reviewed-by: Jeff Davis, Tom
Lane Discussion: https://postgr.es/m/24857.1569775891@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/41a6de41ed697df5d84f3144c6c60b4a9725381f

- Add hooks for session start and session end, take two. These hooks can be used
in loadable modules. A simple test module is included. The first attempt was
done with cd8ce3a but we lacked handling for NO_INSTALLCHECK in the MSVC
scripts (problem solved afterwards by 431f1599) so the buildfarm got angry.
This also fixes a couple of issues noticed upon review compared to the first
attempt, so the code has slightly changed, resulting in a more simple test
module. Author: Fabrízio de Royes Mello, Yugo Nagata Reviewed-by: Andrew
Dunstan, Michael Paquier, Aleksandr Parfenov Discussion:
https://postgr.es/m/20170720204733.40f2b7eb.nagata@sraoss.co.jp Discussion:
https://postgr.es/m/20190823042602.GB5275@paquier.xyz
https://git.postgresql.org/pg/commitdiff/e788bd924c19e296bd34316e30e3ba1b68354e64

- Fix test_session_hooks with parallel workers. Several buildfarm machines have
been complaining about the new module test_session_hooks to be unstable, like
crake and thorntail. The issue was that the module was trying to log some
start and end session activity for parallel workers, which makes little sense
as they don't support DML, so just prevent this pattern to happen in the
module. This could be reproduced by enforcing force_parallel_mode=regress,
which is the value used by some of the buildfarm members. Discussion:
https://postgr.es/m/20191001045246.GF2781@paquier.xyz
https://git.postgresql.org/pg/commitdiff/002962dc7293043126561b0d0df79d6c76251804

- Revert hooks for session start and end, take two. The location of the session
end hook has been chosen so as it is possible to allow modules to do their own
transactions, however any trying to any any subsystem which went through
before_shmem_exit() would cause issues, limiting the pluggability of the hook.
Per discussion with Tom Lane and Andres Freund. Discussion:
https://postgr.es/m/18722.1569906636@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/9555cc8d2b02c4191d67ba39f589b39b01735518

- Remove temporary WAL and history files at the end of archive recovery. cbc55da
has reworked the order of some actions at the end of archive recovery.
Unfortunately this overlooked the fact that the startup process needs to
remove RECOVERYXLOG (for temporary WAL segment newly recovered from archives)
and RECOVERYHISTORY (for temporary history file) at this step, leaving the
files around even after recovery ended. Backpatch to 9.5, like the previous
commit. Author: Sawada Masahiko Reviewed-by: Fujii Masao, Michael Paquier
Discussion:
https://postgr.es/m/CAD21AoBO_eDQub6zojFnWtnmutRBWvYf7=cW4Hsqj+U_R26w3Q@mail.gmail.com
Backpatch-through: 9.5
https://git.postgresql.org/pg/commitdiff/df86e52cace2c4134db51de6665682fb985f3195

- Fix --dry-run mode of pg_rewind. Even if --dry-run mode was specified, the
control file was getting updated, preventing follow-up runs of pg_rewind to
work properly on the target data folder. The origin of the problem came from
the refactoring done by ce6afc6. Author: Alexey Kondratov Discussion:
https://postgr.es/m/7ca88204-3e0b-2f4c-c8af-acadc4b266e5@postgrespro.ru
Backpatch-through: 12
https://git.postgresql.org/pg/commitdiff/6f3823b03560589157d9dbdab623f603ef393d7c

- Fix issues in pg_rewind with --no-ensure-shutdown/--write-recovery-conf. This
fixes two issues with recent features added in pg_rewind: - --dry-run should
do nothing on the target directory, but 927474c forgot to consider that for
--write-recovery-conf. - --no-ensure-shutdown was not actually working. There
is no test coverage for this option yet, but a subsequent patch will add that.
Author: Alexey Kondratov Discussion:
https://postgr.es/m/7ca88204-3e0b-2f4c-c8af-acadc4b266e5@postgrespro.ru
https://git.postgresql.org/pg/commitdiff/6837632b758e0470a2692d9f8303e8aebd6fbd8f

Álvaro Herrera pushed:

- pg_rewind: Allow writing recovery configuration. This is provided with a new
switch --write-recovery-conf and reuses the pg_basebackup code. Author: Paul
Guo, Jimmy Yih, Ashwin Agrawal Reviewed-by: Alexey Kondratov, Michaël Paquier,
Álvaro Herrera Discussion:
https://postgr.es/m/CAEET0ZEffUkXc48pg2iqARQgGRYDiiVxDu+yYek_bTwJF+q=Uw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/927474ce1a2498ddb617c6113a88ca61fbba161d

- pg_rewind: test new --write-recovery-conf functionality. Author: Alexey
Kondratov Reviewed-by: Paul Guo Discussion:
https://postgr.es/m/2f726102-3f1e-bf16-061e-501919473ace@postgrespro.ru
https://git.postgresql.org/pg/commitdiff/7524c788743f387c20bd4719c7a0ef0887602930

Bruce Momjian pushed:

- docs: adjust multi-column most-common-value statistics. This commit adds a
mention that the order of columns specified during multi-column
most-common-value statistics is insignificant, and tries to simplify examples.
Discussion: https://postgr.es/m/20190828162238.GA8360@momjian.us
Backpatch-through: 12
https://git.postgresql.org/pg/commitdiff/7e0fb165dd2447f83464833e63e646d2771edb15

Tomáš Vondra pushed:

- Add transparent block-level memory accounting. Adds accounting of memory
allocated in a memory context. Compared to various ad hoc solutions, the main
advantage is that the accounting is transparent and does not require direct
control over allocations (this matters for use cases where the allocations
happen in user code, like for example aggregate states allocated in a
transition functions). To reduce overhead, the accounting happens at the
block level (not for individual chunks) and only the context immediately
owning the block is updated. When inquiring about amount of memory allocated
in a context, we have to recursively walk all children contexts. This "lazy"
accounting works well for cases with relatively small number of contexts in
the relevant subtree and/or with infrequent inquiries. Author: Jeff Davis
Reivewed-by: Tomas Vondra, Melanie Plageman, Soumyadeep Chakraborty
Discussion:
https://www.postgresql.org/message-id/flat/027a129b8525601c6a680d27ce3a7172dab61aab(dot)camel(at)j-davis(dot)com
https://git.postgresql.org/pg/commitdiff/5dd7fc1519461548eebf26c33eac6878ea3e8788

- Optimize partial TOAST decompression. Commit 4d0e994eed added support for
partial TOAST decompression, so the decompression is interrupted after
producing the requested prefix. For prefix and slices near the beginning of
the entry, this may saves a lot of decompression work. That however only
deals with decompression - the whole compressed entry was still fetched and
re-assembled, even though the compression used only a small fraction of it.
This commit improves that by computing how much compressed data may be needed
to decompress the requested prefix, and then fetches only the necessary part.
We always need to fetch a bit more compressed data than the requested
(uncompressed) prefix, because the prefix may not be compressible at all and
pglz itself adds a bit of overhead. That means this optimization is most
effective when the requested prefix is much smaller than the whole compressed
entry. Author: Binguo Bao Reviewed-by: Andrey Borodin, Tomas Vondra, Paul
Ramsey Discussion:
https://www.postgresql.org/message-id/flat/CAL-OGkthU9Gs7TZchf5OWaL-Gsi=hXqufTxKv9qpNG73d5na_g(at)mail(dot)gmail(dot)com
https://git.postgresql.org/pg/commitdiff/11a078cf87ffb611d19c7dec6df68b41084ad9c9

- Mark two variables in in aset.c with PG_USED_FOR_ASSERTS_ONLY. This fixes two
compiler warnings about unused variables in non-assert builds, introduced by
5dd7fc1519461548eebf26c33eac6878ea3e8788.
https://git.postgresql.org/pg/commitdiff/fa2fe04bf1d4d31e099808745974964f84eb4521

- Blind attempt to fix pglz_maximum_compressed_size. Commit 11a078cf87 triggered
failures on big-endian machines, and the only plausible place for an issue
seems to be that TOAST_COMPRESS_SIZE calls VARSIZE instead of VARSIZE_ANY. So
try fixing that blindly. Discussion:
https://www.postgresql.org/message-id/20191001131803.j6uin7nho7t6vxzy%40development
https://git.postgresql.org/pg/commitdiff/540f31680913b4e11f2caa40cafeca269cfcb22f

- Use Size instead of int64 to track allocated memory. Commit 5dd7fc1519 added
block-level memory accounting, but used int64 variable to track the amount of
allocated memory. That is incorrect, because we have Size for exactly these
purposes, but it was mostly harmless until c477f3e449 which changed how we
handle with repalloc() when downsizing the chunk. Previously we've ignored
these cases and just kept using the original chunk, but now we need to update
the accounting, and the code was doing this: context->mem_allocated +=
blksize - oldblksize; Both blksize and oldblksize are Size (so unsigned)
which means the subtraction underflows, producing a very high positive value.
On 64-bit platforms (where Size has the same size as mem_alllocated) this
happens to work because the result wraps to the right value, but on (some)
32-bit platforms this fails. This fixes two things - it changes mem_allocated
(and related variables) to Size, and it splits the update to two separate
steps, to prevent any underflows. Discussion:
https://www.postgresql.org/message-id/15151.1570163761%40sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/f2369bc610a19563cc00054ccfe9089fac469641

- Change MemoryContextMemAllocated to return Size. Commit f2369bc610 switched
most of the memory accounting from int64 to Size, but it forgot to change the
MemoryContextMemAllocated return type. So this fixes that omission.
Discussion:
https://www.postgresql.org/message-id/11238.1570200198%40sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/36425ece5d6c78177cdc1453a9925a0bb85da59f

Amit Kapila pushed:

- pgbench: add --partitions and --partition-method options. These new options
allow users to partition the pgbench_accounts table by specifying the number
of partitions and partitioning method. The values allowed for partitioning
method are range and hash. This feature allows users to measure the overhead
of partitioning if any. Author: Fabien COELHO Reviewed-by: Amit Kapila, Amit
Langote, Dilip Kumar, Asif Rehman, and Alvaro Herrera Discussion:
https://postgr.es/m/alpine.DEB.2.21.1907230826190.7008@lancre
https://git.postgresql.org/pg/commitdiff/b1c1aa53182372e907f3f7f090e7eb5f432a4c9a

Andrew Gierth pushed:

- Selectively include window frames in expression walks/mutates.
query_tree_walker and query_tree_mutator were skipping the windowClause of the
query, without regard for the fact that the startOffset and endOffset in a
WindowClause node are expression trees that need to be processed. This was an
oversight in commit ec4be2ee6 from 2010 which added the expression fields; the
main symptom is that function parameters in window frame clauses don't work in
inlined functions. Fix (as conservatively as possible since this needs to not
break existing out-of-tree callers) and add tests. Backpatch all the way,
since this has been broken since 9.0. Per report from Alastair McKinley; fix
by me with kibitzing and review from Tom Lane. Discussion:
https://postgr.es/m/DB6PR0202MB2904E7FDDA9D81504D1E8C68E3800@DB6PR0202MB2904.eurprd02.prod.outlook.com
https://git.postgresql.org/pg/commitdiff/b7a1c5539ad34d7357e04cc58f9c02a101482737

Robert Haas pushed:

- Remove AtSubStart_Notify. Allocate notify-related state lazily instead. This
makes trivial subtransactions noticeably faster. Patch by me, reviewed and
tested by Dilip Kumar, Kyotaro Horiguchi, and Jeevan Ladhe. Discussion:
https://postgr.es/m/CA+TgmobE1J22S1eC-6N-je9LgrcwZypkwp+zH6JXo9mc=4Nk3A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/967e276e9f6b485c8577371713a323bf277b6902

- Rename some toasting functions based on whether they are heap-specific. The
old names for the attribute-detoasting functions names included the word
"heap," which seems outdated now that the heap is only one of potentially many
table access methods. On the other hand, toast_insert_or_update and
toast_delete are heap-specific, so rename them by adding "heap_" as a prefix.
Not all of the work of making the TOAST system fully accessible to AMs other
than the heap is done yet, but there seems to be little harm in getting this
renaming out of the way now. Commit 8b94dab06617ef80a0901ab103ebd8754427ef5a
already divided up the functions among various files partially according to
whether it was intended that they should be heap-specific or AM-agnostic, so
this is just clarifying the division contemplated by that commit. Patch by
me, reviewed and tested by Prabhat Sabu, Thomas Munro, Andres Freund, and
Álvaro Herrera. Discussion:
http://postgr.es/m/CA+TgmoZv-=2iWM4jcw5ZhJeL18HF96+W1yJeYrnGMYdkFFnEpQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2e8b6bfa90b252b1e1758364de7deff067d6058a

Noah Misch pushed:

- Report test_atomic_ops() failures consistently, via macros. This prints the
unexpected value in more failure cases, and it removes forty-eight
hand-maintained error messages. Back-patch to 9.5, which introduced these
tests. Reviewed (in an earlier version) by Andres Freund. Discussion:
https://postgr.es/m/20190915160021.GA24376@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/e800bd7414df3ce8170761e5b75b13e83f576988

== Pending Patches ==

Alexander Korotkov sent in two revisions of a patch to fix a deadlock between
ginDeletePage() and ginStepRight(), and fix traversing to the deleted page from
downlink.

Fujii Masao sent in two revisions of a patch to ignore restore command in crash
recovery.

Fujii Masao sent in two revisions of a patch to fix an assertion failure in
latch.

Konstantin Knizhnik sent in another revision of a patch to remove unneeded
self-joins.

Pengzhou Tang sent in another revision of a patch to execute grouping
set queries in parallel.

Dmitry Dolgov sent in another revision of a patch to implement generic type
subscripting.

Yugo Nagata sent in another revision of a patch to implement incremental view
maintenance.

Andrew Dunstan sent in a patch to ensure that msys2 has pexports.

Anastasia Lubennikova sent in a patch to implement an opclass bitwise equality
check.

Rushabh Lathia sent in another revision of a patch to ensure that pg_basebackup
formats time as UTC.

Peter Eisentraut sent in two revisions of a patch to remove the use of the
deprecated Autoconf define, and simplify the PGAC_STRUCT_TIMEZONE Autoconf
macro.

Kyotaro HORIGUCHI sent in a patch to fix inconsistent usage of BACKEND_*
symbols.

Magnus Hagander sent in another revision of a patch to enable checksums online.

Amit Khandekar sent in two more revisions of a patch to implement minimal
logical decoding on standbys.

Mike Palmiotto sent in another revision of a patch to rework auxiliary
processes.

Joe Nelson sent in another revision of a patch to change atoi to strtol in the
same place.

Greg Nancarrow sent in two more revisions of a patch to add an option to libpq
to connect to a standby server as a priority.

Peter Geoghegan sent in another revision of a patch to add deduplication to
nbtree, and add pageinspect instrumentation for debugging same.

John Hsu sent in a patch to include RELKIND_TOASTVALUE in get_relkind_objtype.

Peter Smith sent in three more revisions of a patch to make use of C99
designated initialisers for nulls/values arrays.

Tom Lane sent in a patch to remove bpchar pattern match ops.

Yuya Watari sent in a patch to silence some compiler warnings about implicit
conversion from 'long' to 'double'.

Daisuke Higuchi sent in a patch to fix some cases where ECPG reports
"unsupported feature will be passed to server" when the command is, in fact,
supported.

Fujii Masao sent in another revision of a patch to fix the performance of DROP
DATABASE with many tablespaces.

Vigneshwaran C sent in a patch to fix the ordering of header file inclusion.

Nikita Glukhov sent in another revision of a patch to fix the parsing of
identifiers in jsonpath.

Pavel Stěhule sent in two more revisions of a patch to implement dropdb --force.

Masahiko Sawada sent in two more revisions of a patch to implement block-level
parallel VACUUM.

Noah Misch sent in a patch to remove -qsrcmsg on AIX.

Fujii Masao sent in a patch to add a new developer GUC "ignore_invalid_pages"

Dilip Kumar sent in another revision of a patch to fix some infelicities between
logical_work_mem and logical streaming of large in-progress transactions.

Amit Khandekar sent in another revision of a patch to use vfd for logrep.

Thomas Munro sent in another revision of a patch to use libc version as a
collation version on glibc systems, along with a WIP patch to use querylocale()
for collation versions on FreeBSD.

Michaël Paquier sent in another revision of a patch to add a hook on session
start.

Rob sent in a patch to fix bug 16032: under Windows, when the backup is aborted
or fails, pg_basebackup is unable to cleanup the backup as the filehandles
haven't been released.

Juan José Santamaría Flecha sent in another revision of a patch to fix a bug
that manifested as wrong results using initcap() with non normalized string.

Amit Langote sent in another revision of a patch to fix a bug that manifested as
dropping column prevented due to inherited index.

Asif Rehman and Jeevan Chalke traded patches to implement parallel backup.

Ermilin Sviatoslav sent in a patch to close stderr and stdout in syslogger.

Pavel Stěhule sent in another revision of a patch to implement schema variables.

Antonin Houska sent in another revision of a patch to consolidate the things
that read XLOG pages to a single code path.

Anastasia Lubennikova sent in another revision of a patch to fix an issue that
manifested as pg_upgrade failures with non-standard ACLs.

Peter Eisentraut sent in a patch to remove some code for old unsupported
versions of MSVC.

Robert Haas sent in a patch to allow TOAST tables to be implemented using table
AMs other than heap.

Vigneshwaran C sent in a patch to update some broken web links.

Noah Misch sent in another revision of a patch to fix a bug that manifested as a
deadlock in XLogInsert on AIX.

Nikolay Shaplov sent in a patch to add some useful asserts into ViewOptions
macros.

Nikolay Shaplov sent in two more revisions of a patch to avoid using
StdRdOptions in access methods.

Álvaro Herrera sent in a patch to fix a bug that manifested as parallel restore
failures in cases of FKs to partitioned tables.

Nikolay Shaplov sent in a patch to use separate a PartitionedRelOptions
structure to store partitioned table options.

Matheus de Oliveira sent in another revision of a patch to add ON UPDATE/DELETE
actions in ALTER CONSTRAINT.

Tom Lane sent in a patch to add a check which ensures that there are enough
child slots for new bgworkers.