== PostgreSQL Weekly News - April 7, 2019 ==

== PostgreSQL Weekly News - April 7, 2019 ==

== PostgreSQL Product News ==

psycopg2 2.8, a Python connector for PostgreSQL, released.
http://initd.org/psycopg/articles/2019/04/04/psycopg-28-released/

repmgr 4.3, a replication manager for PostgreSQL, released.
https://repmgr.org/docs/4.3/release-4.3.html

pgAdmin4 4.4, a web- and native GUI control center for PostgreSQL, released.
https://www.pgadmin.org/docs/pgadmin4/dev/release_notes_4_4.html

== PostgreSQL Jobs for April ==

http://archives.postgresql.org/pgsql-jobs/2019-04/

== PostgreSQL Local ==

The German-speaking PostgreSQL Conference 2019 will take place on May 10, 2019
in Leipzig.
http://2019.pgconf.de/

PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy.
https://2019.pgday.it/en/

PGCon 2019 will take place in Ottawa on May 28-31, 2019.
https://www.pgcon.org/2019

Swiss PGDay 2019 will take place in Rapperswil (near Zurich) on June 28, 2019.
The CfP is open through April 18, 2019, and registration is open.
http://www.pgday.ch/2019/

PostgresLondon 2019 will be July 2-3, 2019 with an optional training day on
July 1.
http://postgreslondon.org

PGConf.Brazil 2019 is on August 1-3 2019 in São Paulo.
http://pgconf.com.br

The first Austrian pgDay, will take place September 6, 2019 at the Hilton Garden
Inn in Wiener Neustadt. The CfP is open until April 1, 2019.
https://pgday.at/en/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Compute root->qual_security_level in a less random place. We can set this up
once and for all in subquery_planner's initial survey of the flattened
rangetable, rather than incrementally adjusting it in build_simple_rel. The
previous approach made it rather hard to reason about exactly when the value
would be available, and we were definitely using it in some places before the
final value was computed. Noted while fooling around with Amit Langote's
patch to delay creation of inheritance child rels. That didn't break this
code, but it made it even more fragile, IMO.
https://git.postgresql.org/pg/commitdiff/9fd4de119c609b4110eea74f00f7b9002e14aa38

- Add test case exercising formerly-unreached code in inheritance_planner. There
was some debate about whether the code I'd added to remap AppendRelInfos
obtained from the initial SELECT planning run is actually necessary. Add a
test case demonstrating that it is. Discussion:
https://postgr.es/m/23831.1553873385@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/8fba397f0ca7b9a1fd59ab2b676c057dde4f8219

- Add volatile qualifier missed in commit 2e616dee9. Noted by Pavel Stehule
Discussion:
https://postgr.es/m/CAFj8pRAaGO5FX7bnP3E=mRssoK8y5T78x7jKy-vDiyS68L888Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/b2b819019f4449f2fd889ce0c39ffa81ebfaee16

- Improve documentation about our XML functionality. Add a section explaining
how our XML features depart from current versions of the SQL standard. Update
and clarify the descriptions of some XML functions. Chapman Flack, reviewed
by Ryan Lambert Discussion:
https://postgr.es/m/5BD1284C.1010305@anastigmatix.net Discussion:
https://postgr.es/m/5C81F8C0.6090901@anastigmatix.net Discussion:
https://postgr.es/m/CAN-V+g-6JqUQEQZ55Q3toXEN6d5Ez5uvzL4VR+8KtvJKj31taw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/12d46ac392d00fa0d103f445aa4997f97029e007

- Restrict pgbench's zipfian parameter to ensure good performance. Remove the
code that supported zipfian distribution parameters less than 1.0, as it had
undocumented performance hazards, and it's not clear that the case is useful
enough to justify either fixing or documenting those hazards. Also, since the
code path for parameter > 1.0 could perform badly for values very close to
1.0, establish a minimum allowed value of 1.001. This solution seems superior
to the previous vague documentation warning about small values not performing
well. Fabien Coelho, per a gripe from Tomas Vondra Discussion:
https://postgr.es/m/b5e172e9-ad22-48a3-86a3-589afa20e8f7@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/26a76cb64072df6fa5585c2c15df39970ccdce01

- Remove now-unnecessary thread pointer arguments in pgbench. Not required after
nuking the zipfian thread-local cache. Also add a comment about hazardous
pointer punning in threadRun(), and avoid using "thread" to refer to the
threads array as a whole. Fabien Coelho and Tom Lane, per suggestion from
Alvaro Herrera Discussion:
https://postgr.es/m/alpine.DEB.2.21.1904032126060.7997@lancre
https://git.postgresql.org/pg/commitdiff/d8c0bd9fefa9c70a3f5613fba672fa92f08ea940

- Doc: clarify partial-index example. Jonathan Katz Discussion:
https://postgr.es/m/155432280882.722.12392985690846288230@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/af052daec5aa282eab2d22fa3b24d6109a30c2f3

- Make queries' locking of indexes more consistent. The assertions added by
commit b04aeb0a0 exposed that there are some code paths wherein the executor
will try to open an index without holding any lock on it. We do have some
lock on the index's table, so it seems likely that there's no fatal problem
with this (for instance, the index couldn't get dropped from under us).
Still, it's bad practice and we should fix it. To do so, remove the
optimizations in ExecInitIndexScan and friends that tried to avoid taking a
lock on an index belonging to a target relation, and just take the lock
always. In non-bug cases, this will result in no additional shared-memory
access, since we'll find in the local lock table that we already have a lock
of the desired type; hence, no significant performance degradation should
occur. Also, adjust the planner and executor so that the type of lock taken
on an index is always identical to the type of lock taken for its table, by
relying on the recently added RangeTblEntry.rellockmode field. This avoids
some corner cases where that might not have been true before (possibly
resulting in extra locking overhead), and prevents future maintenance issues
from having multiple bits of logic that all needed to be in sync. In
addition, this change removes all core calls to ExecRelationIsTargetRelation,
which avoids a possible O(N^2) startup penalty for queries with large numbers
of target relations. (We'd probably remove that function altogether, were it
not that we advertise it as something that FDWs might want to use.) Also
adjust some places in selfuncs.c to not take any lock on indexes they are
transiently opening, since we can assume that plancat.c did that already. In
passing, change gin_clean_pending_list() to take RowExclusiveLock not
AccessShareLock on its target index. Although it's not clear that that's
actually a bug, it seemed very strange for a function that's explicitly going
to modify the index to use only AccessShareLock. David Rowley, reviewed by
Julien Rouhaud and Amit Langote, a bit of further tweaking by me Discussion:
https://postgr.es/m/19465.1541636036@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/9c703c169a872d144f2f79d2fb211c82587adfa7

- Add a "SQLSTATE-only" error verbosity option to libpq and psql. This is
intended for use mostly in test scripts for external tools, which could do
without cross-PG-version variations in error message wording. Of course, the
SQLSTATE isn't guaranteed stable either, but it should be more so than the
error message text. Note: there's a bit of an ABI change for libpq here, but
it seems OK because if somebody compiles against a newer version of
libpq-fe.h, and then tries to pass PQERRORS_SQLSTATE to PQsetErrorVerbosity()
of an older libpq library, it will be accepted and then act like
PQERRORS_DEFAULT, thanks to the way the tests in pqBuildErrorMessage3 have
historically been phrased. That seems acceptable. Didier Gautheron, reviewed
by Dagfinn Ilmari Mannsåker Discussion:
https://postgr.es/m/CAJRYxuKyj4zA+JGVrtx8OWAuBfE-_wN4sUMK4H49EuPed=mOBw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7bac3acab4d5c3f2c35aa3a7bea08411d83fd5bc

- Ensure consistent name matching behavior in processSQLNamePattern(). Prior to
v12, if you used a collation-sensitive regex feature in a pattern handled by
processSQLNamePattern() (for instance, \d '\\w+' in psql), the behavior you
got matched the database's default collation. Since commit 586b98fdf you'd
usually get C-collation behavior, because the catalog "name"-type columns are
now marked as COLLATE "C". Add explicit COLLATE specifications to restore the
prior behavior. (Note for whoever writes the v12 release notes: the need for
this shows that while 586b98fdf preserved pre-v12 behavior of "name" columns
for simple comparison operators, it changed the behavior of regex operators on
those columns. Although this patch fixes it for pattern matches generated by
our own tools, user-written queries will still be affected. So we'd better
mention this issue as a compatibility item.) Daniel Vérité Discussion:
https://postgr.es/m/701e51f0-0ec0-4e70-a365-1958d66dd8d2@manitou-mail.org
https://git.postgresql.org/pg/commitdiff/478cacb50e4851760db9b8e75b2256085b571965

- Use Append rather than MergeAppend for scanning ordered partitions. If we need
ordered output from a scan of a partitioned table, but the ordering matches
the partition ordering, then we don't need to use a MergeAppend to combine the
pre-ordered per-partition scan results: a plain Append will produce the same
results. This both saves useless comparison work inside the MergeAppend
proper, and allows us to start returning tuples after istarting up just the
first child node not all of them. However, all is not peaches and cream,
because if some of the child nodes have high startup costs then there will be
big discontinuities in the tuples-returned-versus-elapsed-time curve. The
planner's cost model cannot handle that (yet, anyway). If we model the
Append's startup cost as being just the first child's startup cost, we may
drastically underestimate the cost of fetching slightly more tuples than are
available from the first child. Since we've had bad experiences with
over-optimistic choices of "fast start" plans for ORDER BY LIMIT queries, that
seems scary. As a klugy workaround, set the startup cost estimate for an
ordered Append to be the sum of its children's startup costs (as MergeAppend
would). This doesn't really describe reality, but it's less likely to cause a
bad plan choice than an underestimated startup cost would. In practice, the
cases where we really care about this optimization will have child plans that
are IndexScans with zero startup cost, so that the overly conservative
estimate is still just zero. David Rowley, reviewed by Julien Rouhaud and
Antonin Houska Discussion:
https://postgr.es/m/CAKJS1f-hAqhPLRk_RaSFTgYxd=Tz5hA7kQ2h4-DhJufQk8TGuw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/959d00e9dbe4cfcf4a63bb655ac2c29a5e579246

- Fix failures in validateForeignKeyConstraint's slow path. The
foreign-key-checking loop in ATRewriteTables failed to ignore relations
without storage (e.g., partitioned tables), unlike the initial loop. This
accidentally worked as long as RI_Initial_Check succeeded, which it does in
most practical cases (including all the ones exercised in the existing
regression tests :-(). However, if that failed, as for instance when there
are permissions issues, then we entered the slow
fire-the-trigger-on-each-tuple path. And that would try to read from the
referencing relation, and fail if it lacks storage. A second problem,
recently introduced in HEAD, was that this loop had been broken by sloppy
refactoring for the tableam API changes. Repair both issues, and add a
regression test case so we have some coverage on this code path. Back-patch
as needed to v11. (It looks like this code could do with additional
bulletproofing, but let's get a working test case in place first.) Hadi
Moshayedi, Tom Lane, Andres Freund Discussion:
https://postgr.es/m/CAK=1=WrnNmBbe5D9sm3t0a6dnAq3cdbF1vXY816j1wsMqzC8bw@mail.gmail.com
Discussion: https://postgr.es/m/19030.1554574075@sss.pgh.pa.us Discussion:
https://postgr.es/m/20190325180405.jytoehuzkeozggxx%40alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/46e3442c9ec858071d60a1c0fae2e9868aeaa0c8

- Avoid Python memory leaks in hstore_plpython and jsonb_plpython. Fix some
places where we might fail to do Py_DECREF() on a Python object (thereby
leaking it for the rest of the session). Almost all of the risks were in
error-recovery paths, which we don't really expect to hit anyway. Hence,
while this is definitely a bug fix, it doesn't quite seem worth back-patching.
Nikita Glukhov, Michael Paquier, Tom Lane Discussion:
https://postgr.es/m/28053a7d-10d8-fc23-b05c-b4749c873f63@postgrespro.ru
https://git.postgresql.org/pg/commitdiff/9e360f0e8327bcf4b7c5ce957024fdff8f95f7de

- Clean up side-effects of commits ab5fcf2b0 et al. Before those commits,
partitioning-related code in the executor could assume that
ModifyTableState.resultRelInfo[] contains only leaf partitions. However, now a
fully-pruned update results in a dummy ModifyTable that references the root
partitioned table, and that breaks some stuff. In v11, this led to an
assertion or core dump in the tuple routing code. Fix by disabling tuple
routing, since we don't need that anyway. (I chose to do that in HEAD as well
for safety, even though the problem doesn't manifest in HEAD as it stands.)
In v10, this confused ExecInitModifyTable's decision about whether it needed
to close the root table. But we can get rid of that altogether by being
smarter about where to find the root table. Note that since the referenced
commits haven't shipped yet, this isn't fixing any bug the field has seen.
Amit Langote, per a report from me Discussion:
https://postgr.es/m/20710.1554582479@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/159970bcadbbdefd312d75ce7ad90f19add19b40

Peter Geoghegan pushed:

- Fix nbtree high key "continuescan" row compare bug. Commit 29b64d1d mishandled
skipping over truncated high key attributes during row comparisons. The row
comparison key matching loop would loop forever when a truncated attribute was
encountered for a row compare subkey. Fix by following the example of other
code in the loop: advance the current subkey, or break out of the loop when
the last subkey is reached. Add test coverage for the relevant
_bt_check_rowcompare() code path. The new test case is somewhat tied to nbtree
implementation details, which isn't ideal, but seems unavoidable.
https://git.postgresql.org/pg/commitdiff/76a39f2295ecb040f2ea052320941e1eb9b526c0

- Invalidate binary search bounds consistently. _bt_check_unique() failed to
invalidate binary search bounds in the event of a live conflict following
commit e5adcb78. This resulted in problems after waiting for the conflicting
xact to commit or abort. The subsequent call to _bt_check_unique() would
restore the initial binary search bounds, rather than starting a new search.
Fix by explicitly invalidating bounds when it becomes clear that there is a
live conflict that insertion will have to wait to resolve. Ashutosh Sharma,
with a few additional tweaks by me. Author: Ashutosh Sharma Reported-By:
Ashutosh Sharma Diagnosed-By: Ashutosh Sharma Discussion:
https://postgr.es/m/CAE9k0PnQp-qr-UYKMSCzdC2FBzdE4wKP41hZrZvvP26dKLonLg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/74eb2176bf3ac7a2fae1282b4f4f1d000f3e6d78

- Add test coverage for rootdescend verification. Commit c1afd175, which added
support for rootdescend verification to amcheck, added only minimal regression
test coverage. Address this by making sure that rootdescend verification is
run on a multi-level index. In passing, simplify some of the regression tests
that exercise multi-level nbtree page deletion. Both issues spotted while
rereviewing coverage of the nbtree patch series using gcov.
https://git.postgresql.org/pg/commitdiff/344b7e11bbaf5e11f2497b11405e63d190043cfe

Andres Freund pushed:

- tableam: Formatting and other minor cleanups. The superflous heapam_xlog.h
includes were reported by Peter Geoghegan.
https://git.postgresql.org/pg/commitdiff/4bb50236eb561f4639e75a393a5a1c9b8681acfb

- tableam: sample scan. This moves sample scan support to below tableam. It's
not optional as there is, in contrast to e.g. bitmap heap scans, no
alternative way to perform tablesample queries. If an AM can't deal with the
block based API, it will have to throw an ERROR. The tableam callbacks for
this are block based, but given the current TsmRoutine interface, that seems
to be required. The new interface doesn't require TsmRoutines to perform
visibility checks anymore - that requires the TsmRoutine to know details about
the AM, which we want to avoid. To continue to allow taking the returned
number of tuples account SampleScanState now has a donetuples field (which
previously e.g. existed in SystemRowsSamplerData), which is only incremented
after the visibility check succeeds. Author: Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/73c954d24896aeb05de0f81d75e891a858e439e9

- tableam: bitmap table scan. This moves bitmap heap scan support to below an
optional tableam callback. It's optional as the whole concept of bitmap
heapscans is fairly block specific. This basically moves the work previously
done in bitgetpage() into the new scan_bitmap_next_block callback, and the
direct poking into the buffer done in BitmapHeapNext() into the new
scan_bitmap_next_tuple() callback. The abstraction is currently somewhat
leaky because nodeBitmapHeapscan.c's prefetching and visibilitymap based logic
remains - it's likely that we'll later have to move more into the AM. But it's
not trivial to do so without introducing a significant amount of code
duplication between the AMs, so that's a project for later. Note that now
nodeBitmapHeapscan.c and the associated node types are a bit misnamed. But
it's not clear whether renaming wouldn't be a cure worse than the disease.
Either way, that'd be best done in a separate commit. Author: Andres Freund
Reviewed-By: Robert Haas (in an older version) Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/bfbcad478f05794e5e7ea1339e62a1d258c99b6c

- tableam: Add table_finish_bulk_insert(). This replaces the previous calls of
heap_sync() in places using bulk-insert. By passing in the flags used for
bulk-insert the AM can decide (first at insert time and then during the finish
call) which of the optimizations apply to it, and what operations are
necessary to finish a bulk insert operation. Also change HEAP_INSERT_* flags
to TABLE_INSERT, and rename hi_options to ti_options. These changes are made
even in copy.c, which hasn't yet been converted to tableam. There's no harm in
doing so. Author: Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/d45e40158623baacd3f36f92a670d435cc1e845f

- Only allow heap in a number of contrib modules. Contrib modules pgrowlocks,
pgstattuple and some functionality in pageinspect currently only supports the
heap table AM. As they are all concerned with low-level details that aren't
reasonably exposed via tableam, error out if invoked on a non heap relation.
Author: Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/4b82664156c230b59607704506f5b0a32ef490a2

- tableam: basic documentation. This adds documentation about the user oriented
parts of table access methods (i.e. the default_table_access_method GUC and
the USING clause for CREATE TABLE etc), adds a basic chapter about the table
access method interface, and adds a note to storage.sgml that it's contents
don't necessarily apply for non-builtin AMs. Author: Haribabu Kommi and
Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/b73c3a11963c8bb783993cfffabb09f558f86e37

- tableam: Add table_multi_insert() and revamp/speed-up COPY FROM buffering.
This adds table_multi_insert(), and converts COPY FROM, the only user of
heap_multi_insert, to it. A simple conversion of COPY FROM use slots would
have yielded a slowdown when inserting into a partitioned table for some
workloads. Different partitions might need different slots (both slot types
and their descriptors), and dropping / creating slots when there's constant
partition changes is measurable. Thus instead revamp the COPY FROM buffering
for partitioned tables to allow to buffer inserts into multiple tables,
flushing only when limits are reached across all partition buffers. By only
dropping slots when there've been inserts into too many different partitions,
the aforementioned overhead is gone. By allowing larger batches, even when
there are frequent partition changes, we actuall speed such cases up
significantly. By using slots COPY of very narrow rows into unlogged /
temporary might slow down very slightly (due to the indirect function calls).
Author: David Rowley, Andres Freund, Haribabu Kommi Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://postgr.es/m/20190327054923.t3epfuewxfqdt22e@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/86b85044e823a304d2a265abc030254d39efe7df

- Remove unused struct member, enforce multi_insert callback presence. Author:
David Rowley, Andres Freund Discussion:
https://postgr.es/m/CAKJS1f9=9phmm66diAji4gvHnWSrK7BGFoNct+mEUT_c8pPOjw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/57a7a3adfe4eadb471bb45a7effb1a99ebb00a7b

- Harden tableam against nonexistant / wrong kind of AMs. Previously it was
allowed to set default_table_access_method to an empty string. That makes
sense for default_tablespace, where that was copied from, as it signals
falling back to the database's default tablespace. As there is no equivalent
for table AMs, forbid that. Also make sure to throw a usable error when
creating a table using an index AM, by using get_am_type_oid() to implement
get_table_am_oid() instead of a separate copy. Previously we'd error out only
later, in GetTableAmRoutine(). Thirdly remove GetTableAmRoutineByAmId() - it
was only used in an earlier version of 8586bf7ed8. Add tests for the above
(some for index AMs as well).
https://git.postgresql.org/pg/commitdiff/ea97e440b8570ffd1a6cd6604f2ef882c0a72291

- table: docs: fix typos and grammar. Author: Justin Pryzby Discussion:
https://postgr.es/m/20190404055138.GA24864@telsasoft.com
https://git.postgresql.org/pg/commitdiff/86cc06d1cf9c30be3b79207242e6746f0f0b681c

Noah Misch pushed:

- Update HINT for pre-existing shared memory block. One should almost always
terminate an old process, not use a manual removal tool like ipcrm. Removal
of the ipcclean script eleven years ago
(39627b1ae680cba44f6e56ca5facec4fdbfe9495) and its non-replacement corroborate
that manual shm removal is now a niche goal. Back-patch to 9.4 (all supported
versions). Reviewed by Daniel Gustafsson and Kyotaro HORIGUCHI. Discussion:
https://postgr.es/m/20180812064815.GB2301738@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/5a907404b52753c4d6c6a7c21765aeaa42fd6a3b

- Consistently test for in-use shared memory. postmaster startup scrutinizes any
shared memory segment recorded in postmaster.pid, exiting if that segment
matches the current data directory and has an attached process. When the
postmaster.pid file was missing, a starting postmaster used weaker checks.
Change to use the same checks in both scenarios. This increases the chance of
a startup failure, in lieu of data corruption, if the DBA does "kill -9 `head
-n1 postmaster.pid` && rm postmaster.pid && pg_ctl -w start". A postmaster
will no longer recycle segments pertaining to other data directories. That's
good for production, but it's bad for integration tests that crash a
postmaster and immediately delete its data directory. Such a test now leaks a
segment indefinitely. No "make check-world" test does that. win32_shmem.c
already avoided all these problems. In 9.6 and later, enhance PostgresNode to
facilitate testing. Back-patch to 9.4 (all supported versions). Reviewed by
Daniel Gustafsson and Kyotaro HORIGUCHI. Discussion:
https://postgr.es/m/20130911033341.GD225735@tornado.leadboat.com
https://git.postgresql.org/pg/commitdiff/2f932f71d9f2963bbd201129d7b971c8f5f077fd

- Handle USE_MODULE_DB for all tests able to use an installed postmaster. When
$(MODULES) and $(MODULE_big) are empty, derive the database name from the
first element of $(REGRESS) instead of using a constant string. When deriving
the database name from $(MODULES), use its first element instead of the entire
list; the earlier approach would fail if any multi-module directory had
$(REGRESS) tests. Treat isolation suites and src/pl correspondingly. Under
USE_MODULE_DB=1, installcheck-world and check-world no longer reuse any
database name in a given postmaster. Buildfarm members axolotl, mandrill and
frogfish saw spurious "is being accessed by other users" failures that would
not have happened without database name reuse. (The CountOtherDBBackends() 5s
deadline expired during DROP DATABASE; a backend for an earlier test suite had
used the same database name and had not yet exited.) Back-patch to 9.4 (all
supported versions), except bits pertaining to isolation suites. Concept
reviewed by Andrew Dunstan, Andres Freund and Tom Lane. Discussion:
https://postgr.es/m/20190401135213.GE891537@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/f433394e48afbb733b1436276a5a08cd78443164

- Assert that pgwin32_signal_initialize() has been called early enough. Before
the pgwin32_signal_initialize() call, the backend version of pg_usleep() has
no effect. No in-tree code falls afoul of that today, but temporary commit
23078689a9921968ac0873b017be6e7f772f10bc did so. Discussion:
https://postgr.es/m/20190402135442.GA1173872@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/ab9ed9be2378f02edf613e0543a29287e7484338

- Make src/test/recovery/t/017_shm.pl safe for concurrent execution. Buildfarm
members idiacanthus and komodoensis, which share a host, both executed this
test in the same second. That failed. Back-patch to 9.6, where the test
first appeared. Discussion:
https://postgr.es/m/20190404020543.GA1319573@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/16ee6eaf80a40007a138b60bb5661660058d0422

- Silence -Wimplicit-fallthrough in sysv_shmem.c. Commit
2f932f71d9f2963bbd201129d7b971c8f5f077fd added code that elicits a warning on
buildfarm member flaviventris. Back-patch to 9.4, like that commit. Reported
by Andres Freund. Discussion:
https://postgr.es/m/20190404020057.galelv7by75ekqrh@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/6f0e190056fe441f7cf788ff19b62b13c94f68f3

- Revert "Consistently test for in-use shared memory.". This reverts commits
2f932f71d9f2963bbd201129d7b971c8f5f077fd,
16ee6eaf80a40007a138b60bb5661660058d0422 and
6f0e190056fe441f7cf788ff19b62b13c94f68f3. The buildfarm has revealed several
bugs. Back-patch like the original commits. Discussion:
https://postgr.es/m/20190404145319.GA1720877@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/82150a05be0dc8a8ec708d1f6e6e1ef26c0e58d4

Michaël Paquier pushed:

- Fix thinko in allocation call during MVC list deserialization. Spotted by
Coverity.
https://git.postgresql.org/pg/commitdiff/4ae7f02b0364ccba49a82efbfff46125fb357d6c

- Add progress reporting to pg_checksums. This adds a new option to pg_checksums
called -P/--progress, showing every second some information about the
computation state of an operation for --check and --enable (--disable only
updates the control file and is quick). This requires a pre-scan of the data
folder so as the total size of checksummable items can be calculated, and then
it gets compared to the amount processed. Similarly to what is done for
pg_rewind and pg_basebackup, the information printed in the progress report
consists of the current amount of data computed and the total amount of data
to compute. This could be extended later on. Author: Michael Banck, Bernd
Helmle Reviewed-by: Fabien Coelho, Michael Paquier Discussion:
https://postgr.es/m/1535719851.1286.17.camel@credativ.de
https://git.postgresql.org/pg/commitdiff/280e5f14056bf34a0f52320f659fb93acfda0876

- Improve readability of some tests in strings.sql. c251336 has added some tests
to check if a toast relation should be empty or not, hardcoding the toast
relation name when calling pg_relation_size(). pg_class.reltoastrelid offers
the same information, so simplify the tests to use that. Reviewed-by: Daniel
Gustafsson Discussion: https://postgr.es/m/20190403065949.GH3298@paquier.xyz
https://git.postgresql.org/pg/commitdiff/92c76021aed47daa61afd5c7d6f8201382dee203

- Fix some documentation in pg_rewind. Since 11, it is possible to use a
non-superuser role when using an online source cluster with pg_rewind as long
as the role has proper permissions to execute on the source all the functions
used by pg_rewind, and the documentation stated that a superuser is necessary.
Let's add at the same time all the details needed to create such a role. A
second confusion which comes a lot from users is that it is necessary to issue
a checkpoint on a freshly-promoted standby so as its control file has
up-to-date timeline information which is used by pg_rewind to validate the
operation. Let's document that properly. This is back-patched down to 9.5
where pg_rewind has been introduced. Author: Michael Paquier Reviewed-by:
Magnus Hagander Discussion:
https://postgr.es/m/CABUevEz5bpvbwVsYCaSMV80CBZ5-82nkMzbb+Bu=h1m=rLdn=g@mail.gmail.com
Backpatch-through: 9.5
https://git.postgresql.org/pg/commitdiff/bfc80683ce51590545350b2c8ccfaa9e7a2a1870

- Add support TCP user timeout in libpq and the backend server. Similarly to the
set of parameters for keepalive, a connection parameter for libpq is added as
well as a backend GUC, called tcp_user_timeout. Increasing the TCP user
timeout is useful to allow a connection to survive extended periods without
end-to-end connection, and decreasing it allows application to fail faster.
By default, the parameter is 0, which makes the connection use the system
default, and follows a logic close to the keepalive parameters in its
handling. When connecting through a Unix-socket domain, the parameters have
no effect. Author: Ryohei Nagaura Reviewed-by: Fabien Coelho, Robert Haas,
Kyotaro Horiguchi, Kirk Jamison, Mikalai Keida, Takayuki Tsunakawa, Andrei
Yahorau Discussion:
https://postgr.es/m/EDA4195584F5064680D8130B1CA91C45367328@G01JPEXMBYT04
https://git.postgresql.org/pg/commitdiff/249d64999615802752940e017ee5166e726bc7cd

Peter Eisentraut pushed:

- Catch syntax error in generated column definition. The syntax GENERATED
BY DEFAULT AS (expr) is not allowed but we have to accept it in the grammar
to avoid shift/reduce conflicts because of the similar syntax for identity
columns. The existing code just ignored this, incorrectly. Add an explicit
error check and a bespoke error message. Reported-by: Justin Pryzby
<pryzby(at)telsasoft(dot)com>
https://git.postgresql.org/pg/commitdiff/7241911782a7420e38b6e50b57d304ea48bc5362

- Unified logging system for command-line programs. This unifies the various ad
hoc logging (message printing, error printing) systems used throughout the
command-line programs. Features: - Program name is automatically prefixed.
- Message string does not end with newline. This removes a common source of
inconsistencies and omissions. - Additionally, a final newline is
automatically stripped, simplifying use of PQerrorMessage() etc., another
common source of mistakes. - I converted error message strings to use %m
where possible. - As a result of the above several points, more translatable
message strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences. - There is support for setting a "log level". This is not meant
to be user-facing, but can be used internally to implement debug or
verbose modes. - Lazy argument evaluation, so no significant overhead if
logging at some level is disabled. - Some color in the messages, similar to
gcc and clang. Set PG_COLOR=auto to try it out. Some colors are
predefined, but can be customized by setting PG_COLORS. - Common files
(common/, fe_utils/, etc.) can handle logging much more simply by just using
one API without worrying too much about the context of the calling program,
requiring callbacks, or having to pass "progname" around everywhere. - Some
programs called setvbuf() to make sure that stderr is unbuffered, even on
Windows. But not all programs did that. This is now done centrally. Soft
goals: - Reduces vertical space use and visual complexity of error reporting
in the source code. - Encourages more deliberate classification of messages.
For example, in some cases it wasn't clear without analyzing the surrounding
code whether a message was meant as an error or just an info. - Concepts
and terms are vaguely aligned with popular logging frameworks such as log4j
and Python logging. This is all just about printing stuff out. Nothing
affects program flow (e.g., fatal exits). The uses are just too varied to do
that. Some existing code had wrappers that do some kind of print-and-exit, and
I adapted those. I tried to keep the output mostly the same, but there is a
lot of historical baggage to unwind and special cases to consider, and I might
not always have succeeded. One significant change is that pg_rewind used to
write all error messages to stdout. That is now changed to stderr.
Reviewed-by: Donald Dong <xdong(at)csumb(dot)edu> Reviewed-by: Arthur Zakirov
<a(dot)zakirov(at)postgrespro(dot)ru> Discussion:
https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310(at)2ndquadrant(dot)com
https://git.postgresql.org/pg/commitdiff/cc8d41511721d25d557fc02a46c053c0a602fed0

- file_fdw: Fix for generated columns. Since file_fdw uses COPY internally, but
COPY doesn't allow listing generated columns in its column list, we need to
make sure that we don't add generated columns to the column lists internally
generated by file_fdw. Reported-by: Erik Rijkers <er(at)xs4all(dot)nl>
https://git.postgresql.org/pg/commitdiff/33215d113d61980a4b2f2aec36cdcd21fa58c1c4

- Fix compiler warning. Rewrite get_attgenerated() to avoid compiler warning if
the compiler does not recognize that elog(ERROR) does not return.
Reported-by: David Rowley <david(dot)rowley(at)2ndquadrant(dot)com>
https://git.postgresql.org/pg/commitdiff/edda32ee250fe480701083d4e072e06929991bf9

- Cast pg_stat_progress_cluster.cluster_index_relid to oid. It's tracked
internally as bigint, but when presented to the user it should be oid.
https://git.postgresql.org/pg/commitdiff/106f2eb664bbd38c83090becff9fcde1e9622c9c

- Report progress of REINDEX operations. This uses the same infrastructure that
the CREATE INDEX progress reporting uses. Add a column to
pg_stat_progress_create_index to report the OID of the index being worked on.
This was not necessary for CREATE INDEX, but it's useful for REINDEX. Also
edit the phase descriptions a bit to be more consistent with the source code
comments. Discussion:
https://www.postgresql.org/message-id/ef6a6757-c36a-9e81-123f-13b19e36b7d7%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/03f9e5cba0ee1633af4abe734504df50af46fbd8

Alexander Korotkov pushed:

- GIN support for @@ and @? jsonpath operators. This commit makes existing GIN
operator classes jsonb_ops and json_path_ops support "jsonb @@ jsonpath" and
"jsonb @? jsonpath" operators. Basic idea is to extract statements of
following form out of jsonpath. key1.key2. ... .keyN = const The rest of
jsonpath is rechecked from heap. Catversion is bumped. Discussion:
https://postgr.es/m/fcc6fc6a-b497-f39a-923d-aa34d0c588e8%402ndQuadrant.com
Author: Nikita Glukhov, Alexander Korotkov Reviewed-by: Jonathan Katz, Pavel
Stehule
https://git.postgresql.org/pg/commitdiff/0a02e2ae0236103e641f6570b8135b7ee8a83686

- Restrict some cases in parsing numerics in jsonpath. Jsonpath now accepts
integers with leading zeroes and floats starting with a dot. However, SQL
standard requires to follow JSON specification, which doesn't allow none of
these cases. Our json[b] datatypes also restrict that. So, restrict it in
jsonpath altogether. Author: Nikita Glukhov
https://git.postgresql.org/pg/commitdiff/2e643501e5281ad5e0fa626dab1d51c1d38f639a

- Throw error in jsonb_path_match() when result is not single boolean.
jsonb_path_match() checks if jsonb document matches jsonpath query.
Therefore, jsonpath query should return single boolean. Currently, if result
of jsonpath is not a single boolean, NULL is returned independently whether
silent mode is on or off. But that appears to be wrong when silent mode is
off. This commit makes jsonb_path_match() throw an error in this case.
Author: Nikita Glukhov
https://git.postgresql.org/pg/commitdiff/b4cc19ab01ffe6a72a915b21aa41536de80923f5

Thomas Munro pushed:

- Fix deadlock in heap_compute_xid_horizon_for_tuples(). We can't call code that
uses syscache while we hold buffer locks on a catalog relation. If passed
such a relation, just fall back to the general effective_io_concurrency GUC
rather than trying to look up the containing tablespace's IO concurrency
setting. We might find a better way to control prefetching in follow-up work,
but for now this is enough to avoid the deadlock introduced by commit
558a9165e0. Reviewed-by: Andres Freund Diagnosed-by: Peter Geoghegan
Discussion:
https://postgr.es/m/CA%2BhUKGLCwPF0S4Mk7S8qw%2BDK0Bq65LueN9rofAA3HHSYikW-Zw%40mail.gmail.com
Discussion:
https://postgr.es/m/962831d8-c18d-180d-75fb-8b842e3a2742%40chrullrich.net
https://git.postgresql.org/pg/commitdiff/4fd05bb55b40a3c9dde2b19942f275fc31b5225a

- Add wal_recycle and wal_init_zero GUCs. On at least ZFS, it can be beneficial
to create new WAL files every time and not to bother zero-filling them. Since
it's not clear which other filesystems might benefit from one or both of those
things, add individual GUCs to control those two behaviors independently and
make only very general statements in the docs. Author: Jerry Jelinek, with
some adjustments by Thomas Munro Reviewed-by: Alvaro Herrera, Andres Freund,
Tomas Vondra, Robert Haas and others Discussion:
https://postgr.es/m/CACPQ5Fo00QR7LNAcd1ZjgoBi4y97%2BK760YABs0vQHH5dLdkkMA%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/475861b2615dd63ae8431d811749a6f9a15bbfd6

- Refactor the fsync queue for wider use. Previously, md.c and checkpointer.c
were tightly integrated so that fsync calls could be handed off and processed
in the background. Introduce a system of callbacks and file tags, so that
other modules can hand off fsync work in the same way. For now only md.c uses
the new interface, but other users are being proposed. Since there may be use
cases that are not strictly SMGR implementations, use a new function table for
sync handlers rather than extending the traditional SMGR one. Instead of
using a bitmapset of segment numbers for each RelFileNode in the
checkpointer's hash table, make the segment number part of the key. This
requires sending explicit "forget" requests for every segment individually
when relations are dropped, but suits the file layout schemes of proposed
future users better (ie sparse or high segment numbers). Author: Shawn
Debnath and Thomas Munro Reviewed-by: Thomas Munro, Andres Freund Discussion:
https://postgr.es/m/CAEepm=2gTANm=e3ARnJT=n0h8hf88wqmaZxk0JYkxw+b21fNrw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3eb77eba5a51780d5cf52cd66a9844cd4d26feb0

- Fix bugs in mdsyncfiletag(). Commit 3eb77eba moved a _mdfd_getseg() call from
mdsync() into a new callback function mdsyncfiletag(), but didn't get the
arguments quite right. Without the EXTENSION_DONT_CHECK_SIZE flag we fail to
open a segment if lower-numbered segments have been truncated, and it wants a
block number rather than a segment number. While comparing with the older
coding, also remove an unnecessary clobbering of errno, and adjust the code in
mdunlinkfiletag() to ressemble the original code from mdpostckpt() more
closely instead of using an unnecessary call to smgropen(). Author: Thomas
Munro Discussion:
https://postgr.es/m/CA%2BhUKGL%2BYLUOA0eYiBXBfwW%2BbH5kFgh94%3DgQH0jHEJ-t5Y91wQ%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/794c543b1736f71ff5bc5231eee41cdf460ad8ce

- Wake up interested backends when a checkpoint fails. Commit c6c9474a switched
to condition variables instead of sleep loops to notify backends of checkpoint
start and stop, but forgot to broadcast in case of checkpoint failure.
Author: Thomas Munro Discussion:
https://postgr.es/m/CA%2BhUKGJKbCd%2B_K%2BSEBsbHxVT60SG0ivWHHAdvL0bLTUt2xpA2w%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/de2b38419c4018fc8c2c917cc2cc752063096cfb

Dean Rasheed pushed:

- Perform RLS subquery checks as the right user when going via a view. When
accessing a table with RLS via a view, the RLS checks are performed as the
view owner. However, the code neglected to propagate that to any subqueries in
the RLS checks. Fix that by calling setRuleCheckAsUser() for all RLS policy
quals and withCheckOption checks for RTEs with RLS. Back-patch to 9.5 where
RLS was added. Per bug #15708 from daurnimator. Discussion:
https://postgr.es/m/15708-d65cab2ce9b1717a@postgresql.org
https://git.postgresql.org/pg/commitdiff/e2d28c0f404713f564dc2250646551c75172f17b

Etsuro Fujita pushed:

- postgres_fdw: Perform the (ORDERED, NULL) upperrel operations remotely. The
upper-planner pathification allows FDWs to arrange to push down different
types of upper-stage operations to the remote side. This commit teaches
postgres_fdw to do it for the (ORDERED, NULL) upperrel, which is responsible
for evaluating the query's ORDER BY ordering. Since postgres_fdw is already
able to evaluate that ordering remotely for foreign baserels and foreign
joinrels (see commit aa09cd242f et al.), this adds support for that for
foreign grouping relations. Author: Etsuro Fujita Reviewed-By: Antonin Houska
and Jeff Janes Discussion:
https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/ffab494a4d4676a71e830b2d7285eeeb5f2b53d7

- postgres_fdw: Modify regression tests for EPQ-related planning problems. This
prevents the tests added by commit 4bbf6edfbd and adjusted by commit
99f6a17dd6 from being useless by plan changes created by an upcoming commit.
Author: Etsuro Fujita Discussion:
https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/0269edefacb78700c95572c41a5a4c8633b631b6

- Refactor create_limit_path() to share cost adjustment code with FDWs. This is
in preparation for an upcoming commit. Author: Etsuro Fujita Reviewed-By:
Antonin Houska and Jeff Janes Discussion:
https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/aef65db6769e3f2c855dd89edcf95a536a9ab74f

- postgres_fdw: Perform the (FINAL, NULL) upperrel operations remotely. The
upper-planner pathification allows FDWs to arrange to push down different
types of upper-stage operations to the remote side. This commit teaches
postgres_fdw to do it for the (FINAL, NULL) upperrel, which is responsible for
doing LockRows, LIMIT, and/or ModifyTable. This provides the ability for
postgres_fdw to handle SELECT commands so that it 1) skips the LockRows step
(if any) (note that this is safe since it performs early locking) and 2)
pushes down the LIMIT and/or OFFSET restrictions (if any) to the remote side.
This doesn't handle the INSERT/UPDATE/DELETE cases. Author: Etsuro Fujita
Reviewed-By: Antonin Houska and Jeff Janes Discussion:
https://postgr.es/m/87pnz1aby9.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/d50d172e517c1d2aabff3ceb3ad3113b909c5017

- Doc: Update documentation on partitioning vs. foreign tables. The limitations
that it is not allowed to create/attach a foreign table as a partition of an
indexed partitioned table were not documented. Reported-By: Stepan Yankevych
Author: Etsuro Fujita Reviewed-By: Amit Langote Backpatch-through: 11 where
partitioned index was introduced Discussion:
https://postgr.es/m/1553869152.858391073.5f8m3n0x@frv53.fwdcdn.com
https://git.postgresql.org/pg/commitdiff/3e6b0c472942e9a73e0fc10659dce2cba56959a7

Stephen Frost pushed:

- Add support for partial TOAST decompression. When asked for a slice of a TOAST
entry, decompress enough to return the slice instead of decompressing the
entire object. For use cases where the slice is at, or near, the beginning of
the entry, this avoids a lot of unnecessary decompression work. This changes
the signature of pglz_decompress() by adding a boolean to indicate if it's ok
for the call to finish before consuming all of the source or destination
buffers. Author: Paul Ramsey Reviewed-By: Rafia Sabih, Darafei Praliaskouski,
Regina Obe Discussion:
https://postgr.es/m/CACowWR07EDm7Y4m2kbhN_jnys%3DBBf9A6768RyQdKm_%3DNpkcaWg%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/4d0e994eed83c845a05da6e9a417b4efec67efaf

- GSSAPI encryption support. On both the frontend and backend, prepare for
GSSAPI encryption support by moving common code for error handling into a
separate file. Fix a TODO for handling multiple status messages in the
process. Eliminate the OIDs, which have not been needed for some time. Add
frontend and backend encryption support functions. Keep the context
initiation for authentication-only separate on both the frontend and backend
in order to avoid concerns about changing the requested flags to include
encryption support. In postmaster, pull GSSAPI authorization checking into a
shared function. Also share the initiator name between the encryption and
non-encryption codepaths. For HBA, add "hostgssenc" and "hostnogssenc"
entries that behave similarly to their SSL counterparts. "hostgssenc"
requires either "gss", "trust", or "reject" for its authentication.
Similarly, add a "gssencmode" parameter to libpq. Supported values are
"disable", "require", and "prefer". Notably, negotiation will only be
attempted if credentials can be acquired. Move credential acquisition into
its own function to support this behavior. Add a simple pg_stat_gssapi view
similar to pg_stat_ssl, for monitoring if GSSAPI authentication was used, what
principal was used, and if encryption is being used on the connection.
Finally, add documentation for everything new, and update existing
documentation on connection security. Thanks to Michael Paquier for the
Windows fixes. Author: Robbie Harwood, with changes to the read/write
functions by me. Reviewed in various forms and at different times by: Michael
Paquier, Andres Freund, David Steele. Discussion:
https://www.postgresql.org/message-id/flat/jlg1tgq1ktm(dot)fsf(at)thriss(dot)redhat(dot)com
https://git.postgresql.org/pg/commitdiff/b0b39f72b9904bcb80f97b35837ccff1578aa4b8

- Move the be_gssapi_get_* prototypes. The be_gssapi_get_* prototypes were put
close to similar ones for SSL- but a bit too close since that meant they ended
up only being included for SSL-enabled builds. Move those to be under
ENABLE_GSS instead. Pointed out by Tom.
https://git.postgresql.org/pg/commitdiff/87e16db5ebe7256712999ec690899237029b9ae7

- Handle errors during GSSAPI startup better. There was some confusion over the
format of the error message returned from the server during GSSAPI startup;
specifically, it was expected that a length would be returned when, in
reality, at this early stage in the startup sequence, no length is returned
from the server as part of an error message. Correct the client-side code for
dealing with error messages sent by the server during startup by simply
reading what's available into our buffer, after we've discovered it's an error
message, and then reporting back what was returned. In passing, also add in
documentation of the environment variable PGGSSENCMODE which was missed
previously, and adjust the code to look for the PGGSSENCMODE variable (the
environment variable change was missed in the prior GSSMODE -> GSSENCMODE
commit). Error-handling issue discovered by Peter Eisentraut, the rest were
items discovered during testing of the error handling.
https://git.postgresql.org/pg/commitdiff/c46c85d4594d52fb34d36d4761bb9cfc5626f20b

Álvaro Herrera pushed:

- Report progress of CREATE INDEX operations. This uses the progress reporting
infrastructure added by c16dc1aca5e0, adding support for CREATE INDEX and
CREATE INDEX CONCURRENTLY. There are two pieces to this: one is
index-AM-agnostic, and the other is AM-specific. The latter is fairly
elaborate for btrees, including reportage for parallel index builds and the
separate phases that btree index creation uses; other index AMs, which are
much simpler in their building procedures, have simplistic reporting only, but
that seems sufficient, at least for non-concurrent builds. The
index-AM-agnostic part is fairly complete, providing insight into the
CONCURRENTLY wait phases as well as block-based progress during the index
validation table scan. (The index validation index scan requires patching
each AM, which has not been included here.) Reviewers: Rahila Syed, Pavan
Deolasee, Tatsuro Yamada Discussion:
https://postgr.es/m/20181220220022.mg63bhk26zdpvmcj@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/ab0dfc961b6a821f23d9c40c723d11380ce195a6

- Update expected output for modified catalog definition. Pilot error in
previous commit
https://git.postgresql.org/pg/commitdiff/11074f26bc072334ab91e5646d113f95f884bd07

- Prevent use of uninitialized variable. Per buildfarm member longfin.
https://git.postgresql.org/pg/commitdiff/e8abf97af770401934a2fc4887940b76403520f0

- Correctly initialize newly added struct member. Valgrind was rightly
complaining that IndexVacuumInfo->report_progress (added by commit
ab0dfc961b6a) was not being initialized in some code paths. Repair. Per
buildfarm member lousyjack.
https://git.postgresql.org/pg/commitdiff/5f768045a1a25847a3eea59d309e28c8141aed44

- Support foreign keys that reference partitioned tables. Previously, while
primary keys could be made on partitioned tables, it was not possible to
define foreign keys that reference those primary keys. Now it is possible to
do that. Author: Álvaro Herrera Reviewed-by: Amit Langote, Jesper Pedersen
Discussion: https://postgr.es/m/20181102234158.735b3fevta63msbj@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/f56f8f8da6afd8523b4d5284e02a20ed2b33ef8d

- Copy name when cloning FKs recurses to partitions. We were passing a string
owned by a syscache entry, which was released before recursing. Fix by
pstrdup'ing the string. Per buildfarm member prion.
https://git.postgresql.org/pg/commitdiff/5f6fc34af5712995026267b9ace79d06b258a0ae

- Log all statements from a sample of transactions. This is useful to obtain a
view of the different transaction types in an application, regardless of the
durations of the statements each runs. Author: Adrien Nayrat Reviewed-by:
Masahiko Sawada, Hayato Kuroda, Andres Freund
https://git.postgresql.org/pg/commitdiff/799e220346f1387e823a4dbdc3b1c8c3cdc5c3e0

- Tweak docs for log_statement_sample_rate. Author: Justin Pryzby, partly after
a suggestion from Masahiko Sawada Discussion:
https://postgr.es/m/20190328135918.GA27808@telsasoft.com Discussion:
https://postgr.es/m/CAD21AoB9+y8N4+Fan-ne-_7J5yTybPttxeVKfwUocKp4zT1vNQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d1f04b96b99d595e80791cdb0faa9cfdde2a5afa

- pg_restore: Require "-f -" to mean stdout. The previous convention that stdout
was selected by default when nothing is specified was just too error-prone.
After a suggestion from Andrew Gierth. Author: Euler Taveira Reviewed-by:
Yoshikazu Imai, José Arthur Benetasso Villanova Discussion:
https://postgr.es/m/87sgwrmhdv.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/413ccaa74d9a126b042727c826e65e2844adadac

- Add facility to copy replication slots. This allows the user to create
duplicates of existing replication slots, either logical or physical, and even
changing properties such as whether they are temporary or the output plugin
used. There are multiple uses for this, such as initializing multiple
replicas using the slot for one base backup; when doing investigation of
logical replication issues; and to select a different output plugins. Author:
Masahiko Sawada Reviewed-by: Michael Paquier, Andres Freund, Petr Jelinek
Discussion:
https://postgr.es/m/CAD21AoAm7XX8y_tOPP6j4Nzzch12FvA1wPqiO690RCk+uYVstg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/9f06d79ef831ffa333f908f6d3debdb654292414

- psql \dP: list partitioned tables and indexes The new command lists
partitioned relations (tables and/or indexes), possibly with their sizes,
possibly including partitioned partitions; their parents (if not top-level);
if indexes show the tables they belong to; and their descriptions. While
there are various possible improvements to this, having it in this form is
already a great improvement over not having any way to obtain this report.
Author: Pavel Stěhule, with help from Mathias Brossard, Amit Langote and
Justin Pryzby. Reviewed-by: Amit Langote, Mathias Brossard, Melanie Plageman,
Michaël Paquier, Álvaro Herrera
https://git.postgresql.org/pg/commitdiff/1c5d9270e339662cdd78d51d0b859d4f0a11aa91

Heikki Linnakangas pushed:

- Generate less WAL during GiST, GIN and SP-GiST index build. Instead of
WAL-logging every modification during the build separately, first build the
index without any WAL-logging, and make a separate pass through the index at
the end, to write all pages to the WAL. This significantly reduces the amount
of WAL generated, and is usually also faster, despite the extra I/O needed for
the extra scan through the index. WAL generated this way is also faster to
replay. For GiST, the LSN-NSN interlock makes this a little tricky. All pages
must be marked with a valid (i.e. non-zero) LSN, so that the parent-child
LSN-NSN interlock works correctly. We now use magic value 1 for that during
index build. Change the fake LSN counter to begin from 1000, so that 1 is
safely smaller than any real or fake LSN. 2 would've been enough for our
purposes, but let's reserve a bigger range, in case we need more special
values in the future. Author: Anastasia Lubennikova, Andrey V. Lepikhov
Reviewed-by: Heikki Linnakangas, Dmitry Dolgov
https://git.postgresql.org/pg/commitdiff/9155580fd5fc2a0cbb23376dfca7cd21f59c2c7b

Tomáš Vondra pushed:

- Reduce overhead of pg_mcv_list (de)serialization. Commit ea4e1c0e8f resolved
issues with memory alignment in serialized pg_mcv_list values, but it required
copying data to/from the varlena buffer during serialization and
deserialization. As the MCV lits may be fairly large, the overhead (memory
consumption, CPU usage) can get rather significant too. This change tweaks
the serialization format so that the alignment is correct with respect to the
varlena value, and so the parts may be accessed directly without copying the
data. Catversion bump, as it affects existing pg_statistic_ext data.
https://git.postgresql.org/pg/commitdiff/c50b3158bfd7fc8c2a0b2d2ede4415fd9859acd1

- Add SETTINGS option to EXPLAIN, to print modified settings. Query planning is
affected by a number of configuration options, and it may be crucial to know
which of those options were set to non-default values. With this patch you
can say EXPLAIN (SETTINGS ON) to include that information in the query plan.
Only options affecting planning, with values different from the built-in
default are printed. This patch also adds auto_explain.log_settings option,
providing the same capability in auto_explain module. Author: Tomas Vondra
Reviewed-by: Rafia Sabih, John Naylor Discussion:
https://postgr.es/m/e1791b4c-df9c-be02-edc5-7c8874944be0@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/ea569d64ac7174d3fe657e3e682d11053ecf1866

Robert Haas pushed:

- Allow VACUUM to be run with index cleanup disabled. This commit adds a new
reloption, vacuum_index_cleanup, which controls whether index cleanup is
performed for a particular relation by default. It also adds a new option to
the VACUUM command, INDEX_CLEANUP, which can be used to override the
reloption. If neither the reloption nor the VACUUM option is used, the
default is true, as before. Masahiko Sawada, reviewed and tested by Nathan
Bossart, Alvaro Herrera, Kyotaro Horiguchi, Darafei Praliaskouski, and me. The
wording of the documentation is mostly due to me. Discussion:
http://postgr.es/m/CAD21AoAt5R3DNUZSjOoXDUY=naYPUOuffVsRzuTYMz29yLzQCA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/a96c41feec6b6616eb9d5baee9a9e08c20533c38

- Fix missing word. Nathan Bossart Discussion:
http://postgr.es/m/2C63765B-AD31-4F6C-8DA7-C8544634C714@amazon.com
https://git.postgresql.org/pg/commitdiff/6665305e17edff3e2c43d71396ea5b6a69e7b6f3

== Pending Patches ==

Raymond Martin sent in two more revisions of a patch to make pg_stat_statements
more efficient.

Amit Langote and Tom Lane traded patches to speed up planning with partitions.

Surafel Temesgen and Tomáš Vondra traded patches to implement FETCH FIRST ...
WITH TIES

Peter Geoghegan sent in a patch to test the recent patch to make all B-Trees
unique by making TID part of the comparisons.

Pavel Stěhule sent in another revision of a patch to add type info support
functions for functions that use "any" type.

Pavel Stěhule sent in another revision of a patch to add schema variables.

Kyotaro HORIGUCHI sent in two more revisions of a patch to fix the WAL-skipping
feature.

Yuzuko Hosoya sent in two more revisions of a patch to fix a problem with
default partition pruning.

Noah Misch sent in a patch to fix some mysterious crashes on Windows.

Julien Rouhaud sent in three more revisions of a patch to fix a checksum error
in pg_stat_databases.

Liu Huailing sent in a patch to fix a memory overflow in the ECPG preproc
module.

Konstantin Knizhnik sent in another revision of a patch to add autoprepare.

Nikolay Shaplov sent in another revision of a patch to add a dummy index am.

Pavan Deolasee sent in another revision of a patch to implement COPY ... FREEZE.

Liu Huailing sent in a patch to fix some misspellings.

Amit Khandekar sent in two more revisions of a patch to implement minimal
logical decoding on standbys.

Michaël Paquier sent in another revision of a patch to add a separate
table-level option to control compression.

Haribabu Kommi and Amit Kapila traded patches to avoid counting transaction
stats for parallel worker cooperating transaction.

Kyotaro HORIGUCHI sent in a patch to fix a confusing message when starting
standby without setting restore_command.

Konstantin Knizhnik sent in a patch to test a combination of inheritance,
invalidations, and prepared statements.

Daniel Gustafsson sent in a patch to ensure that binaries used in pg_upgrade are
all the same minor version.

Dmitry Dolgov sent in another revision of a patch to optimize GROUP BY.

Heikki Linnakangas sent in a patch to refactor the checks for deleted GiST pages
and use the full 64-bit XID for checking if a deleted GiST page is old enough.

Kyotaro HORIGUCHI sent in three more revisions of a patch to protect syscache
from bloating with negative cache entries.

Michaël Paquier sent in a patch to fix a bug in toast tuple target.

Masahiko Sawada sent in three more revisions of a patch to implement block-level
parallel vacuum.

Daniel Vérité sent in a patch to fix an infelicity introduced by making the
collation of the catalogs 'C'.

Justin Pryzby sent in another revision of a patch to fix query logging of
prepared statements.

Takayuki Tsunakawa and Fujii Masao traded patches to add a reloption to prevent
VACUUM from truncating empty pages at the end of relation.

Iwata Aya sent in another revision of a patch to implement a libpq debug log.

Anastasia Lubennikova sent in a patch to add a test to reproduce gist index
problem caused by patch Generate less WAL during GiST, GIN and SP-GiST index
build.

Artur Zakirov sent in another revision of a patch to implement shared ispell
dictionaries.

Andres Freund sent in another revision of a patch to convert GiST to compute
page level XID horizon on the primary.

Nikolay Shaplov sent in another revision of a patch to improve the Ltree syntax.

Pierre Ducroquet sent in a patch to use a different permission check path for
indexes and relations in pg_stats.

Fabien COELHO sent in a patch to add some more control on the initialization
phase of pgbench.

Jose Luis Tallon sent in two revisions of a patch to implement uuid_version().

Fabien COELHO sent in two revisions of a patch to pgbench to add minimal stats
on initialization.

Ramanarayana sent in another revision of a patch to fix a problem during Windows
service start.

Noah Misch sent in a patch to augment every test postgresql.conf.

David Rowley sent in three more revisions of a patch to shrink a bloated
locallocktable.

Julien Rouhaud sent in a patch to remove some trailing whitespace in the
documentation.

Ramanarayana sent in another revision of a patch to add Perldoc for TestLib
module in Postgres.

Thomas Munro sent in a patch to fix pgindent's postprocessing of multi-line
prototypes.

Justin Pryzby sent in another revision of a patch to clean up pg_checksums.sgml.

Justin Pryzby sent in a patch to clean up/remove/update references to OID column.

Andres Freund sent in a PoC patch to fix an infelicity between COPY FREEZE and
setting PD_ALL_VISIBLE/visibility map bits.