== PostgreSQL Weekly News - August 16, 2020 ==

== PostgreSQL Weekly News - August 16, 2020 ==

PostgreSQL security releases 12.4, 11.9, 10.14, 9.6.19, 9.5.23, 13beta3 are out.
Upgrade as soon as possible.
https://www.postgresql.org/about/news/2060/

Person of the week: https://postgresql.life/post/michael_brewer/

== PostgreSQL Product News ==

PostGIS 3.0.2, 2.5.5, and 2.4.9, the industry standard geographic information
system package for PostgreSQL, released.
http://postgis.net/2020/08/15/postgis-patches/

== PostgreSQL Jobs for August ==

http://archives.postgresql.org/pgsql-jobs/2020-08/

== PostgreSQL Local ==

pgDay Israel 2020 will take place on September 10, 2020 in Tel Aviv.
http://pgday.org.il/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Peter Geoghegan pushed:

- Correct nbtree page split lock coupling comment. There is no reason to
distinguish between readers and writers here.
https://git.postgresql.org/pg/commitdiff/d129c07499dbf0d5960115173515e3ce384c662a

- Fix obsolete comment in xlogutils.c. Oversight in commit 2c03216d831.
https://git.postgresql.org/pg/commitdiff/914140e85a79c63853c86334afa2d7e6e930c11a

Tom Lane pushed:

- Make contrib modules' installation scripts more secure. Hostile objects
located within the installation-time search_path could capture references in
an extension's installation or upgrade script. If the extension is being
installed with superuser privileges, this opens the door to privilege
escalation. While such hazards have existed all along, their urgency
increases with the v13 "trusted extensions" feature, because that lets a
non-superuser control the installation path for a superuser-privileged script.
Therefore, make a number of changes to make such situations more secure: *
Tweak the construction of the installation-time search_path to ensure that
references to objects in pg_catalog can't be subverted; and explicitly add
pg_temp to the end of the path to prevent attacks using temporary objects. *
Disable check_function_bodies within installation/upgrade scripts, so that any
security gaps in SQL-language or PL-language function bodies cannot create a
risk of unwanted installation-time code execution. * Adjust lookup of type
input/receive functions and join estimator functions to complain if there are
multiple candidate functions. This prevents capture of references to
functions whose signature is not the first one checked; and it's arguably more
user-friendly anyway. * Modify various contrib upgrade scripts to ensure that
catalog modification queries are executed with secure search paths. (These
are in-place modifications with no extension version changes, since it is the
update process itself that is at issue, not the end result.) Extensions that
depend on other extensions cannot be made fully secure by these methods alone;
therefore, revert the "trusted" marking that commit eb67623c9 applied to
earthdistance and hstore_plperl, pending some better solution to that set of
issues. Also add documentation around these issues, to help extension authors
write secure installation scripts. Patch by me, following an observation by
Andres Freund; thanks to Noah Misch for review. Security: CVE-2020-14350
https://git.postgresql.org/pg/commitdiff/7eeb1d9861b0a3f453f8b31c7648396cdd7f1e59

- Improve comments for postmaster.c's BackendList. This had gotten a little
disjointed over time, and some of the grammar was sloppy. Rewrite for more
clarity. In passing, re-pgindent some recently added comments. No code
changes.
https://git.postgresql.org/pg/commitdiff/3546cf8a7a9dc57e6aa98f5fc1ac5476ad6b99ff

- Doc: improve examples for json_populate_record() and related functions. Make
these examples self-contained by providing declarations of the user-defined
row types they rely on. There wasn't room to do this in the old doc format,
but now there is, and I think it makes the examples a good bit less confusing.
https://git.postgresql.org/pg/commitdiff/a9306f10b95992ec7229cae3de507a9fa2f6aa3c

- Fix postmaster's behavior during smart shutdown. Up to now, upon receipt of a
SIGTERM ("smart shutdown" command), the postmaster has immediately killed all
"optional" background processes, and subsequently refused to launch new ones
while it's waiting for foreground client processes to exit. No doubt this
seemed like an OK policy at some point; but it's a pretty bad one now, because
it makes for a seriously degraded environment for the remaining clients: *
Parallel queries are killed, and new ones fail to launch. (And our
parallel-query infrastructure utterly fails to deal with the case in a
reasonable way --- it just hangs waiting for workers that are not going to
arrive. There is more work needed in that area IMO.) * Autovacuum ceases to
function. We can tolerate that for awhile, but if bulk-update queries
continue to run in the surviving client sessions, there's eventually going to
be a mess. In the worst case the system could reach a forced shutdown to
prevent XID wraparound. * The bgwriter and walwriter are also stopped
immediately, likely resulting in performance degradation. Hence, let's
rearrange things so that the only immediate change in behavior is refusing to
let in new normal connections. Once the last normal connection is gone, shut
everything down as though we'd received a "fast" shutdown. To implement this,
remove the PM_WAIT_BACKUP and PM_WAIT_READONLY states, instead staying in
PM_RUN or PM_HOT_STANDBY while normal connections remain. A subsidiary state
variable tracks whether or not we're letting in new connections in those
states. This also allows having just one copy of the logic for killing child
processes in smart and fast shutdown modes. I moved that logic into
PostmasterStateMachine() by inventing a new state PM_STOP_BACKENDS.
Back-patch to 9.6 where parallel query was added. In principle this'd be a
good idea in 9.5 as well, but the risk/reward ratio is not as good there,
since lack of autovacuum is not a problem during typical uses of smart
shutdown. Per report from Bharath Rupireddy. Patch by me, reviewed by Thomas
Munro Discussion:
https://postgr.es/m/CALj2ACXAZ5vKxT9P7P89D87i3MDO9bfS+_bjMHgnWJs8uwUOOw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/0038f943878286ce84b2dfac10d64e00eab02edd

- Be more careful about the shape of hashable subplan clauses. nodeSubplan.c
expects that the testexpr for a hashable ANY SubPlan has the form of one or
more OpExprs whose LHS is an expression of the outer query's, while the RHS is
an expression over Params representing output columns of the subquery.
However, the planner only went as far as verifying that the clauses were all
binary OpExprs. This works 99.99% of the time, because the clauses have the
right shape when emitted by the parser --- but it's possible for function
inlining to break that, as reported by PegoraroF10. To fix, teach the planner
to check that the LHS and RHS contain the right things, or more accurately
don't contain the wrong things. Given that this has been broken for years
without anyone noticing, it seems sufficient to just give up hashing when it
happens, rather than go to the trouble of commuting the clauses back again
(which wouldn't necessarily work anyway). While poking at that, I also
noticed that nodeSubplan.c had a baked-in assumption that the number of hash
clauses is identical to the number of subquery output columns. Again, that's
fine as far as parser output goes, but it's not hard to break it via function
inlining. There seems little reason for that assumption though --- AFAICS,
the only thing it's buying us is not having to store the number of hash
clauses explicitly. Adding code to the planner to reject such cases would
take more code than getting nodeSubplan.c to cope, so I fixed it that way.
This has been broken for as long as we've had hashable SubPlans, so back-patch
to all supported branches. Discussion:
https://postgr.es/m/1549209182255-0.post@n3.nabble.com
https://git.postgresql.org/pg/commitdiff/1e7629d2c95ffd290ab0e18d7618ca9d9da94265

- Remove no-longer-usable hstore--1.0--1.1.sql update script. Since commit
865f14a2d made "=>" unusable as an operator name, it's been impossible either
to install hstore 1.0 or to execute this update script. There's not much
point in continuing to ship it. Discussion:
https://postgr.es/m/653936.1597431032@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/d4d443b3bbbb3eb9cdc511564ef3c57fde7dd3ac

- Doc: various improvements for pg_basebackup reference page. Put the -r option
in the right section (it certainly isn't an option controlling "the location
and format of the output"). Clarify the behavior of the tablespace and waldir
options (that part per gripe from robert(at)interactive(dot)co(dot)uk). Make a large
number of small copy-editing fixes in text that visibly wasn't written by
native speakers, and try to avoid grammatical inconsistencies between the
descriptions of the different options. Back-patch to v13, since HEAD hasn't
meaningfully diverged yet. Discussion:
https://postgr.es/m/159749418850.14322.216503677134569752@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/db659a3416b967d716806e558efbb9d1ec610cd1

Noah Misch pushed:

- Empty search_path in logical replication apply worker and walsender. This is
like CVE-2018-1058 commit 582edc369cdbd348d68441fc50fa26a84afd0c1a. Today, a
malicious user of a publisher or subscriber database can invoke arbitrary SQL
functions under an identity running replication, often a superuser. This fix
may cause "does not exist" or "no schema has been selected to create in"
errors in a replication process. After upgrading, consider watching server
logs for these errors. Objects accruing schema qualification in the wake of
the earlier commit are unlikely to need further correction. Back-patch to v10,
which introduced logical replication. Security: CVE-2020-14349
https://git.postgresql.org/pg/commitdiff/11da97024abbe76b8c81e3f2375b2a62e9717c67

- Move connect.h from fe_utils to src/include/common. Any libpq client can use
the header. Clients include backend components postgres_fdw, dblink, and
logical replication apply worker. Back-patch to v10, because another fix
needs this. In released branches, just copy the header and keep the original.
https://git.postgresql.org/pg/commitdiff/e078fb5d4eeb23d0d09932e0b183a8e7bdfb17b4

- Document clashes between logical replication and untrusted users. Back-patch
to v10, which introduced logical replication. Security: CVE-2020-14349
https://git.postgresql.org/pg/commitdiff/cec57b1a0fbcd3833086ba686897c5883e0a2afc

- Prevent concurrent SimpleLruTruncate() for any given SLRU. The
SimpleLruTruncate() header comment states the new coding rule. To achieve
this, add locktype "frozenid" and two LWLocks. This closes a rare opportunity
for data loss, which manifested as "apparent wraparound" or "could not access
status of transaction" errors. Data loss is more likely in pg_multixact, due
to released branches' thin margin between multiStopLimit and multiWrapLimit.
If a user's physical replication primary logged ": apparent wraparound"
messages, the user should rebuild standbys of that primary regardless of
symptoms. At less risk is a cluster having emitted "not accepting commands"
errors or "must be vacuumed" warnings at some point. One can test a cluster
for this data loss by running VACUUM FREEZE in every database. Back-patch to
9.5 (all supported versions). Discussion:
https://postgr.es/m/20190218073103.GA1434723@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/566372b3d6435639e4cc4476d79b8505a0297c87

- Correct several behavior descriptions in comments. Reuse cautionary language
from src/test/ssl/README in src/test/kerberos/README. SLRUs have had access
to six-character segments names since commit
73c986adde5d73a5e2555da9b5c8facedb146dcd, and recovery stopped calling
HeapTupleHeaderAdvanceLatestRemovedXid() in commit
558a9165e081d1936573e5a7d576f5febd7fb55a. The other corrections are more
self-evident.
https://git.postgresql.org/pg/commitdiff/676a9c3cc4b5f1d262c29de318868948513f0fa0

Peter Eisentraut pushed:

- Replace remaining StrNCpy() by strlcpy(). They are equivalent, except that
StrNCpy() zero-fills the entire destination buffer instead of providing just
one trailing zero. For all but a tiny number of callers, that's just overhead
rather than being desirable. Remove StrNCpy() as it is now unused. In some
cases, namestrcpy() is the more appropriate function to use. While we're here,
simplify the API of namestrcpy(): Remove the return value, don't check for
NULL input. Nothing was using that anyway. Also, remove a few unused
name-related functions. Reviewed-by: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> Discussion:
https://www.postgresql.org/message-id/flat/44f5e198-36f6-6cdb-7fa9-60e34784daae%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/1784f278a63866cc144fcd0a2127cadba6a2b7f8

- Remove obsolete cygwin.h hack. The version being checked for is 20 years old.
Reviewed-by: Marco Atzeri <marco(dot)atzeri(at)gmail(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/aa266ede-baaa-f4e6-06cf-5b1737610e9a%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/53095b5fe650270118bc2ab77416d08e19472cd3

- Remove obsolete HAVE_BUGGY_SOLARIS_STRTOD. Fixed more than 10 years ago.
Reviewed-by: Noah Misch <noah(at)leadboat(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/aa266ede-baaa-f4e6-06cf-5b1737610e9a%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/bacda6a327efb820d0e9f3262b81e803b2d5702b

Michaël Paquier pushed:

- Improve tab completion of REINDEX in psql. This allows the tab completion of
REINDEX to handle an optional parenthesized list of options. This case is
more complicated than VACUUM or ANALYZE because of CONCURRENTLY and the
different object types to consider with the reindex. Author: Justin Pryzby
Reviewed-by: Alexey Kondratov, Michael Paquier Discussion:
https://postgr.es/m/20200403182712.GR14618@telsasoft.com
https://git.postgresql.org/pg/commitdiff/1f75b454134cce6a67a9bcdb01b5c018221dd359

- Fix compilation warnings with libselinux 3.1 in contrib/sepgsql/. Upstream
SELinux has recently marked security_context_t as officially deprecated,
causing warnings with -Wdeprecated-declarations. This is considered as legacy
code for some time now by upstream as security_context_t got removed from most
of the code tree during the development of 2.3 back in 2014. This removes all
the references to security_context_t in sepgsql/ to be consistent with
SELinux, fixing the warnings. Note that this does not impact the minimum
version of libselinux supported. Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/20200813012735.GC11663@paquier.xyz
https://git.postgresql.org/pg/commitdiff/1f32136a9960df2e135e7b36363ea1f087b514a0

Andres Freund pushed:

- Rename VariableCacheData.nextFullXid to nextXid. Including Full in variable
names duplicates the type information and leads to overly long names. As
FullTransactionId cannot accidentally be casted to TransactionId that does not
seem necessary. Author: Andres Freund Discussion:
https://postgr.es/m/20200724011143.jccsyvsvymuiqfxu@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/fea10a64340e529805609126740a540c8f9daab4

- Track latest completed xid as a FullTransactionId. The reason for doing so is
that a subsequent commit will need that to avoid wraparound issues. As the
subsequent change is large this was split out for easier review. The reason
this is not a perfect straight-forward change is that we do not want track
64bit xids in the procarray or the WAL. Therefore we need to advance
lastestCompletedXid in relation to 32 bit xids. The code for that is now
centralized in MaintainLatestCompletedXid*. Author: Andres Freund
Reviewed-By: Thomas Munro, Robert Haas, David Rowley Discussion:
https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/3bd7f9969a240827bc2effa399170b7565238fd2

- snapshot scalability: Don't compute global horizons while building snapshots.
To make GetSnapshotData() more scalable, it cannot not look at at each proc's
xmin: While snapshot contents do not need to change whenever a read-only
transaction commits or a snapshot is released, a proc's xmin is modified in
those cases. The frequency of xmin modifications leads to, particularly on
higher core count systems, many cache misses inside GetSnapshotData(), despite
the data underlying a snapshot not changing. That is the most significant
source of GetSnapshotData() scaling poorly on larger systems. Without
accessing xmins, GetSnapshotData() cannot calculate accurate horizons /
thresholds as it has so far. But we don't really have to: The horizons don't
actually change that much between GetSnapshotData() calls. Nor are the
horizons actually used every time a snapshot is built. The trick this commit
introduces is to delay computation of accurate horizons until there use and
using horizon boundaries to determine whether accurate horizons need to be
computed. The use of RecentGlobal[Data]Xmin to decide whether a row version
could be removed has been replaces with new GlobalVisTest* functions. These
use two thresholds to determine whether a row can be pruned: 1)
definitely_needed, indicating that rows deleted by XIDs >= definitely_needed
are definitely still visible. 2) maybe_needed, indicating that rows deleted by
XIDs < maybe_needed can definitely be removed GetSnapshotData() updates
definitely_needed to be the xmin of the computed snapshot. When testing
whether a row can be removed (with GlobalVisTestIsRemovableXid()) and the
tested XID falls in between the two (i.e. XID >= maybe_needed && XID <
definitely_needed) the boundaries can be recomputed to be more accurate. As it
is not cheap to compute accurate boundaries, we limit the number of times that
happens in short succession. As the boundaries used by
GlobalVisTestIsRemovableXid() are never reset (with maybe_needed updated by
GetSnapshotData()), it is likely that further test can benefit from an earlier
computation of accurate horizons. To avoid regressing performance when
old_snapshot_threshold is set (as that requires an accurate horizon to be
computed), heap_page_prune_opt() doesn't unconditionally call
TransactionIdLimitedForOldSnapshots() anymore. Both the computation of the
limited horizon, and the triggering of errors (with
SetOldSnapshotThresholdTimestamp()) is now only done when necessary to remove
tuples. This commit just removes the accesses to PGXACT->xmin from
GetSnapshotData(), but other members of PGXACT residing in the same cache line
are accessed. Therefore this in itself does not result in a significant
improvement. Subsequent commits will take advantage of the fact that
GetSnapshotData() now does not need to access xmins anymore. Note: This
contains a workaround in heap_page_prune_opt() to keep the snapshot_too_old
tests working. While that workaround is ugly, the tests currently are not
meaningful, and it seems best to address them separately. Author: Andres
Freund <andres(at)anarazel(dot)de> Reviewed-By: Robert Haas <robertmhaas(at)gmail(dot)com>
Reviewed-By: Thomas Munro <thomas(dot)munro(at)gmail(dot)com> Reviewed-By: David Rowley
<dgrowleyml(at)gmail(dot)com> Discussion:
https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/dc7420c2c9274a283779ec19718d2d16323640c0

- Fix out-of-date version reference, grammar. Time appears to be passing fast.
Reported-By: Peter Geoghegan <pg(at)bowt(dot)ie>
https://git.postgresql.org/pg/commitdiff/b8443eae72b5c36e6b443a2f09b9c605c61a589d

- snapshot scalability: Move PGXACT->xmin back to PGPROC. Now that xmin isn't
needed for GetSnapshotData() anymore, it leads to unnecessary cacheline
ping-pong to have it in PGXACT, as it is updated considerably more frequently
than the other PGXACT members. After the changes in dc7420c2c92, this is a
very straight-forward change. For highly concurrent, snapshot acquisition
heavy, workloads this change alone can significantly increase scalability.
E.g. plain pgbench on a smaller 2 socket machine gains 1.07x for read-only
pgbench, 1.22x for read-only pgbench when submitting queries in batches of
100, and 2.85x for batches of 100 'SELECT';. The latter numbers are obviously
not to be expected in the real-world, but micro-benchmark the snapshot
computation scalability (previously spending ~80% of the time in
GetSnapshotData()). Author: Andres Freund <andres(at)anarazel(dot)de> Reviewed-By:
Robert Haas <robertmhaas(at)gmail(dot)com> Reviewed-By: Thomas Munro
<thomas(dot)munro(at)gmail(dot)com> Reviewed-By: David Rowley <dgrowleyml(at)gmail(dot)com>
Discussion:
https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/1f51c17c68d05c28d5b9294d8013cb9e7e653160

- snapshot scalability: Introduce dense array of in-progress xids. The new array
contains the xids for all connected backends / in-use PGPROC entries in a
dense manner (in contrast to the PGPROC/PGXACT arrays which can have unused
entries interspersed). This improves performance because GetSnapshotData()
always needs to scan the xids of all live procarray entries and now there's no
need to go through the procArray->pgprocnos indirection anymore. As the set
of running top-level xids changes rarely, compared to the number of snapshots
taken, this substantially increases the likelihood of most data required for a
snapshot being in l2 cache. In read-mostly workloads scanning the xids[]
array will sufficient to build a snapshot, as most backends will not have an
xid assigned. To keep the xid array dense ProcArrayRemove() needs to move
entries behind the to-be-removed proc's one further up in the array. Obviously
moving array entries cannot happen while a backend sets it xid. I.e. locking
needs to prevent that array entries are moved while a backend modifies its
xid. To avoid locking ProcArrayLock in GetNewTransactionId() - a fairly hot
spot already - ProcArrayAdd() / ProcArrayRemove() now needs to hold XidGenLock
in addition to ProcArrayLock. Adding / Removing a procarray entry is not a
very frequent operation, even taking 2PC into account. Due to the above, the
dense array entries can only be read or modified while holding ProcArrayLock
and/or XidGenLock. This prevents a concurrent ProcArrayRemove() from shifting
the dense array while it is accessed concurrently. While the new dense array
is very good when needing to look at all xids it is less suitable when
accessing a single backend's xid. In particular it would be problematic to
have to acquire a lock to access a backend's own xid. Therefore a backend's
xid is not just stored in the dense array, but also in PGPROC. This also
allows a backend to only access the shared xid value when the backend had
acquired an xid. The infrastructure added in this commit will be used for the
remaining PGXACT fields in subsequent commits. They are kept separate to make
review easier. Author: Andres Freund <andres(at)anarazel(dot)de> Reviewed-By: Robert
Haas <robertmhaas(at)gmail(dot)com> Reviewed-By: Thomas Munro
<thomas(dot)munro(at)gmail(dot)com> Reviewed-By: David Rowley <dgrowleyml(at)gmail(dot)com>
Discussion:
https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/941697c3c1ae5d6ee153065adb96e1e63ee11224

- snapshot scalability: Move PGXACT->vacuumFlags to ProcGlobal->vacuumFlags.
Similar to the previous commit this increases the chance that data frequently
needed by GetSnapshotData() stays in l2 cache. As we now take care to not
unnecessarily write to ProcGlobal->vacuumFlags, there should be very few
modifications to the ProcGlobal->vacuumFlags array. Author: Andres Freund
<andres(at)anarazel(dot)de> Reviewed-By: Robert Haas <robertmhaas(at)gmail(dot)com>
Reviewed-By: Thomas Munro <thomas(dot)munro(at)gmail(dot)com> Reviewed-By: David Rowley
<dgrowleyml(at)gmail(dot)com> Discussion:
https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/5788e258bb26495fab65ff3aa486268d1c50b123

- snapshot scalability: Move subxact info to ProcGlobal, remove PGXACT. Similar
to the previous changes this increases the chance that data frequently needed
by GetSnapshotData() stays in l2 cache. In many workloads subtransactions are
very rare, and this makes the check for that considerably cheaper. As this
removes the last member of PGXACT, there is no need to keep it around anymore.
On a larger 2 socket machine this and the two preceding commits result in a
~1.07x performance increase in read-only pgbench. For read-heavy mixed r/w
workloads without row level contention, I see about 1.1x. Author: Andres
Freund <andres(at)anarazel(dot)de> Reviewed-By: Robert Haas <robertmhaas(at)gmail(dot)com>
Reviewed-By: Thomas Munro <thomas(dot)munro(at)gmail(dot)com> Reviewed-By: David Rowley
<dgrowleyml(at)gmail(dot)com> Discussion:
https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/73487a60fc1063ba4b5178b69aee4ee210c182c4

Álvaro Herrera pushed:

- BRIN: Handle concurrent desummarization properly. If a page range is
desummarized at just the right time concurrently with an index walk, BRIN
would raise an error indicating index corruption. This is scary and unhelpful;
silently returning that the page range is not summarized is sufficient
reaction. This bug was introduced by commit 975ad4e602ff as additional
protection against a bug whose actual fix was elsewhere. Backpatch equally.
Reported-By: Anastasia Lubennikova <a(dot)lubennikova(at)postgrespro(dot)ru>
Diagnosed-By: Alexander Lakhin <exclusion(at)gmail(dot)com> Discussion:
https://postgr.es/m/2588667e-d07d-7e10-74e2-7e1e46194491@postgrespro.ru
Backpatch: 9.5 - master
https://git.postgresql.org/pg/commitdiff/1f42d35a1d6144a23602b2c0bc7f97f3046cf890

- Handle new HOT chains in index-build table scans. When a table is scanned by
heapam_index_build_range_scan (née IndexBuildHeapScan) and the table lock
being held allows concurrent data changes, it is possible for new HOT chains
to sprout in a page that were unknown when the scan of a page happened. This
leads to an error such as ERROR: failed to find parent tuple for heap-only
tuple at (X,Y) in table "tbl" because the root tuple was not present when we
first obtained the list of the page's root tuples. This can be fixed by
re-obtaining the list of root tuples, if we see that a heap-only tuple appears
to point to a non-existing root. This was reported by Anastasia as occurring
for BRIN summarization (which exists since 9.5), but I think it could
theoretically also happen with CREATE INDEX CONCURRENTLY (much older) or
REINDEX CONCURRENTLY (very recent). It seems a happy coincidence that BRIN
forces us to backpatch this all the way to 9.5. Reported-by: Anastasia
Lubennikova <a(dot)lubennikova(at)postgrespro(dot)ru> Diagnosed-by: Anastasia Lubennikova
<a(dot)lubennikova(at)postgrespro(dot)ru> Co-authored-by: Anastasia Lubennikova
<a(dot)lubennikova(at)postgrespro(dot)ru> Co-authored-by: Álvaro Herrera
<alvherre(at)alvh(dot)no-ip(dot)org> Discussion:
https://postgr.es/m/602d8487-f0b2-5486-0088-0f372b2549fa@postgrespro.ru
Backpatch: 9.5 - master
https://git.postgresql.org/pg/commitdiff/a811ea5bde2fbf450095994b5726dcbf64d68668

- pg_dump: fix dependencies on FKs to partitioned tables. Parallel-restoring a
foreign key that references a partitioned table with several levels of
partitions can fail: pg_restore: while PROCESSING TOC: pg_restore: from TOC
entry 6684; 2606 29166 FK CONSTRAINT fk fk_a_fkey postgres pg_restore: error:
could not execute query: ERROR: there is no unique constraint matching given
keys for referenced table "pk" Command was: ALTER TABLE fkpart3.fk ADD
CONSTRAINT fk_a_fkey FOREIGN KEY (a) REFERENCES fkpart3.pk(a); This happens
in parallel restore mode because some index partitions aren't yet attached to
the topmost partitioned index that the FK uses, and so the index is still
invalid. The current code marks the FK as dependent on the first level of
index-attach dump objects; the bug is fixed by recursively marking the FK on
their children. Backpatch to 12, where FKs to partitioned tables were
introduced. Reported-by: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> Author: Álvaro Herrera
<alvherre(at)alvh(dot)no-ip(dot)org> Discussion:
https://postgr.es/m/3170626.1594842723@sss.pgh.pa.us Backpatch: 12-master
https://git.postgresql.org/pg/commitdiff/2ba5b2db7943742e100834d99548c5d2661a105b

Heikki Linnakangas pushed:

- Fix typo in test comment.
https://git.postgresql.org/pg/commitdiff/5bdf694568ef0b9eebef32002a9ebc1918dd0b4b

Amit Kapila pushed:

- Mark a few logical decoding related variables with PGDLLIMPORT. Commit
7259736a6e added two variables CheckXidAlive and bsysscan to detect concurrent
aborts and used these in inline functions that are part of the API that can be
used by extensions. So it is better to mark them with PGDLLIMPORT.
Reported-by: Thomas Munro Discussion:
https://postgr.es/m/CA+hUKGJ7+HYupd=Pz9+QrXa-C_YnbC4rAYu8V+=OKg=UgdzMeg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/b48cac3b10a02fea2bed684469dd4d36a6616405

== Pending Patches ==

Movead Li sent in another revision of a patch to implement CSN-based snapshots.

Asim Praveen sent in another revision of a patch to start the WAL receiver
before the startup process replays existing WAL.

Surafel Temesgen sent in another revision of a patch to implement FETCH FIRST
... PERCENT.

Bertrand Drouvot sent in another revision of a patch to add a
log_recovery_conflicts_resolution GUC.

Bharath Rupireddy sent in another revision of a patch to to include instructions
on safe usage of cancel_before_shmem_exit() in its comments.

Bertrand Drouvot sent in another revision of a patch to add more detail to the
way the rm_redo_error_callback works.

Pavel Borisov sent in five more revisions of a patch to implement covering
SP-GiST indexes.

Justin Pryzby and Michaël Paquier traded patches to implement REINDEX of
partitioned tables/indexes, allow CREATE INDEX CONCURRENTLY on partitioned
tables, and implement CLUSTER on partitioned tables.

Justin Pryzby sent in another revision of a patch to allow CLUSTER, VACUUM FULL
and REINDEX to change tablespace on the fly.

Álvaro Herrera sent in a patch to flag CREATE INDEX CONCURRENTLY to avoid
spurious waiting.

Mark Wong and Michaël Paquier traded patches to add PL/sample to
src/test/modules/.

Li Japin sent in a patch to add an idle_session_timeout GUC.

Peter Eisentraut sent in another revision of a patch to add more tests for
extract() of date type, expose the internal function for converting int64 to
numeric, and change the return type of EXTRACT to numeric.

Zeng Wenjing sent in another revision of a patch to implement global temporary
tables.

Dmitry Dolgov sent in a patch to reduce ArrayExpr to a const array, and limit
jumbling for repeating constants.

Thomas Munro sent in two revisions of a patch to defer flushing of SLRU files.

Ashutosh Sharma sent in another revision of a patch to add contrib/pg_surgery to
perform surgery on a damaged heap table.

Peter Eisentraut sent in a patch to use pg_get_functiondef() in pg_dump.

Michaël Paquier sent in a patch to fix an old bug in the logic of
ReindexMultipleTables(): the list of relations to process is built in a first
transaction, and then each table is done in an independent transaction, but we
don't actually check that the relation still exists when doing the real work.

Thomas Munro sent in another revision of a patch to implement WAL prefetch.

Dilip Kumar sent in another revision of a patch to add some infrastructure for
custom compression methods and use same to add one (zlib).

Julien Rouhaud sent in two more revisions of a patch to implement collation
versioning.

Mikhail Titov sent in two revisions of a patch to allow DEFAULT in the VALUES
RTE for generated columns.

Amit Kapila and Dilip Kumar traded patches to fix an infelicity between
logical_work_mem and logical streaming of large in-progress transactions.

Masahiko Sawada sent in a patch to clarify the documentation for high
availability.

Heikki Linnakangas sent in a patch to fix printing the last progress report line
in client programs.

Vigneshwaran C sent in another revision of a patch to parallelize COPY.

Thomas Munro sent in three more revisions of a patch to fix
old_snapshot_threshold's time->xid mapping.

Amit Kapila sent in another revision of a patch to add additional information in
vacuum errcontext.

Heikki Linnakangas sent in a patch to make xact.h usable in frontend, and mark
commit and abort WAL records with XLR_SPECIAL_REL_UPDATE.

Dmitry Dolgov sent in another revision of a patch to implement index skip scans.

Paul A Jungwirth sent in another revision of a patch to implement multi-ranges.