== PostgreSQL Weekly News - November 11, 2018 ==

== PostgreSQL Weekly News - November 11, 2018 ==

PostgreSQL security releases 11.1, 10.6, 9.6.11, 9.5.15, 9.4.20, and 9.3.25 released.
This is the final release of the 9.3 series. Please read the announcement below
and upgrade as soon as possible.
https://www.postgresql.org/about/news/1905/

== PostgreSQL Product News ==

pg_back 1.7, a backup script for PostgreSQL, released.
https://github.com/orgrim/pg_back

Bucardo 5.5.0, a replication system with dual-master capability, released.
http://bucardo.org/wiki/Bucardo

PipelineDB v1.0.0 released as an extension for PostgreSQL 11
https://www.pipelinedb.com/blog/high-performance-time-series-aggregation-for-postgresql-11

== PostgreSQL Jobs for November ==

http://archives.postgresql.org/pgsql-jobs/2018-11/

== PostgreSQL Local ==

pgday.Seoul 2018 will be held in Seoul, South Korea on November 3, 2018 in
Seoul. Korean language information is here:
http://pgday.postgresql.kr/

PGDay Argentina 2018 will be held in Buenos Aires on November 21, 2018.
http://www.pgday.com.ar/buenosaires2018/?lang=en

2Q PGConf will be on December 4-5, 2018 in Chicago, IL.
http://www.2qpgconf.com/

PGConf.ASIA 2018 will take place on December 10-12, 2018 in Akihabara, Tokyo,
Japan.
http://www.pgconf.asia/EN/2018/

FOSDEM PGDay 2019, a one day conference held before the main FOSDEM event will
be held in Brussels, Belgium, on Feb 1st, 2019. The CfP is open until November
9th, 2018 at https://2019.fosdempgday.org/callforpapers/ .
https://2019.fosdempgday.org/

Prague PostgreSQL Developer Day 2019 (P2D2 2019) is a two-day
conference that will be held on February 13-14, 2019 in Prague, Czech Republic.
The CfP is open until January 4, 2018 at https://p2d2.cz/callforpapers
http://www.p2d2.cz/

PGConf India 2019 will be on February 13-15, 2019 in Bengaluru, Karnataka.
Proposals are due via https://goo.gl/forms/F9hRjOIsaNasVOAz2 by October 31st, 2017.
http://pgconf.in/

pgDay Paris 2019 will be held in Paris, France on March 12, 2019
at 199bis rue Saint-Martin. The CfP is open until November 30, 2018.
http://2019.pgday.paris/callforpapers/

PGConf APAC 2019 will be held in Singapore March 19-21, 2019. The CfP is open
at http://2019.pgconfapac.org/cfp through November 16, 2018.
http://2019.pgconfapac.org/

PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy. The CfP
is open at https://2019.pgday.it/en/blog/cfp and the Call for Workshops is at
https://2019.pgday.it/en/blog/cfw until January 15, 2019.
https://2019.pgday.it/en/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Michaël Paquier pushed:

- Ignore partitioned tables when processing ON COMMIT DELETE ROWS. Those tables
have no physical storage, making this option unusable with partition trees as
at commit time an actual truncation was attempted. There are still issues
with the way ON COMMIT actions are done when mixing several action types,
however this impacts as well inheritance trees, so this issue will be dealt
with later. Reported-by: Rajkumar Raghuwanshi Author: Amit Langote
Reviewed-by: Michael Paquier, Tom Lane Discussion:
https://postgr.es/m/CAKcux6mhgcjSiB_egqEAEFgX462QZtncU8QCAJ2HZwM-wWGVew@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/4bc772e2afa55f26734ff3fbdf27601db030b7e5

- Block creation of partitions with open references to its parent. When a
partition is created as part of a trigger processing, it is possible that the
partition which just gets created changes the properties of the table the
executor of the ongoing command relies on, causing a subsequent crash. This
has been found possible when for example using a BEFORE INSERT which creates a
new partition for a partitioned table being inserted to. Any attempt to do so
is blocked when working on a partition, with regression tests added for both
CREATE TABLE PARTITION OF and ALTER TABLE ATTACH PARTITION. Reported-by:
Dmitry Shalashov Author: Amit Langote Reviewed-by: Michael Paquier, Tom Lane
Discussion: https://postgr.es/m/15437-3fe01ee66bd1bae1@postgresql.org
Backpatch-through: 10
https://git.postgresql.org/pg/commitdiff/dc3e436b191a8f8d6f35fad952dd3dc314ccabf9

- Switch pg_promote to be parallel-safe. pg_promote uses nothing relying on a
global state, so it is fine to mark it as parallel-safe, conclusion based on a
detailed analysis from Robert Haas. This also fixes an inconsistency where
pg_proc.dat missed to mark the function with its previous value for
proparallel, update which does not matter now as the default is used. Based
on a discussion between multiple folks: Laurenz Albe, Robert Haas, Amit
Kapila, Tom Lane and myself. Discussion:
https://postgr.es/m/20181029082530.GL14242@paquier.xyz
https://git.postgresql.org/pg/commitdiff/8f045e242b2308473f5de532590454294e3d5683

- Reorganize format options of psql in alphabetical order. This makes the
addition of new formats easier, and documentation lookups easier. Author:
Daniel Vérité Reviewed-by: Fabien Coelho Discussion:
https://postgr.es/m/alpine.DEB.2.20.1803081004241.2916@lancre
https://git.postgresql.org/pg/commitdiff/add9182e59084faf57bb264fefcd39fbc93fb116

- Fix incorrect routine name reference in partprune.c. Author: Yuzuko Hosoya
Discussion: https://postgr.es/m/00ac01d4774c$7feac860$7fc05920$@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/170dccc69ddb5eb379ebb5282d699738177d4543

- Fix dependency handling of partitions and inheritance for ON COMMIT This
commit fixes a set of issues with ON COMMIT actions when used on partitioned
tables and tables with inheritance children: - Applying ON COMMIT DROP on a
partitioned table with partitions or on a table with inheritance children
caused a failure at commit time, with complains about the children being
already dropped as all relations are dropped one at the same time. - Applying
ON COMMIT DELETE on a partition relying on a partitioned. table which uses ON
COMMIT DROP would cause the partition truncation to fail as the parent is
removed first. The solution to the first problem is to handle the removal of
all the dependencies in one go instead of dropping relations one-by-one, based
on a suggestion from Álvaro Herrera. So instead all the relation OIDs to
remove are gathered and then processed in one round of multiple deletions.
The solution to the second problem is to reorder the actions, with truncation
happening first and relation drop done after. Even if it means that a
partition could be first truncated, then immediately dropped if its
partitioned table is dropped, this has the merit to keep the code simple as
there is no need to do existence checks on the relations to drop. Contrary to
a manual TRUNCATE on a partitioned table, ON COMMIT DELETE does not cascade to
its partitions. The ON COMMIT action defined on each partition gets the
priority. Author: Michael Paquier Reviewed-by: Amit Langote, Álvaro Herrera,
Robert Haas Discussion:
https://postgr.es/m/68f17907-ec98-1192-f99f-8011400517f5@lab.ntt.co.jp
Backpatch-through: 10
https://git.postgresql.org/pg/commitdiff/319a8101804f3b62512fdce1a3af1c839344b593

- Fix incorrect routine name in xlog_heapam.h.
s/xl_heap_delete/xl_heap_truncate/ in a comment block referring to flags for
truncation. Discussion: https://postgr.es/m/20180413034734.GE1552@paquier.xyz
https://git.postgresql.org/pg/commitdiff/3ce12018948b2c4ead9a47df7ae0b5ffa2fe06d0

Tom Lane pushed:

- Remove unreferenced pg_opfamily entry. The entry with OID 4035, for GIST
jsonb_ops, is unused; apparently it was added in preparation for index support
that never materialized. Remove it, and add a regression test case to detect
future mistakes of the same kind. Discussion:
https://postgr.es/m/17188.1541379745@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/55f3d1029676ad53621716fd15faed5848c9f4cd

- Last-minute updates for release notes. I removed the item about the
pg_stat_statements change from release-11.sgml, as part of a sweep to delete
items already committed in 11.0; but actually we'd best keep it to ensure that
people who've pg_upgraded their databases will take the requisite action.
Also make said action more visible by making it into its own para. Noted by
Jonathan Katz.
https://git.postgresql.org/pg/commitdiff/b0a1ff8a6a01163f830490e6c542ff852d240690

- Remove useless symbol from Makefile.global. I added HAVE_IPV6 to
Makefile.global way back in commit 7703e55c3 so that we could transmit its
value to the shell-script version of initdb. Since initdb was rewritten in C,
it's been finding that out from pg_config.h instead, so this is useless.
Keeping it here just wastes configure and make cycles, plus it's a potential
two-sources-of-truth problem.
https://git.postgresql.org/pg/commitdiff/8f623bedfb4fee5a125b25720e5451379cf26ff9

- Rename rbtree.c functions to use "rbt" prefix not "rb" prefix. The "rb"
prefix is used by Ruby, so that our existing code results in name collisions
that break plruby. We discussed ways to prevent that by adjusting dynamic
linker options, but it seems that at best we'd move the pain to other cases.
Renaming to avoid the collision is the only portable fix anyway. Fortunately,
our rbtree code is not (yet?) widely used --- in core, there's only a single
usage in GIN --- so it seems likely that we can get away with a rename. I
chose to do this basically as s/rb/rbt/g, except for places where there
already was a "t" after "rb". The patch could have been made smaller by only
touching linker-visible symbols, but it would have resulted in oddly
inconsistent-looking code. Better to make it look like "rbt" was the plan all
along. Back-patch to v10. The rbtree.c code exists back to 9.5, but
rb_iterate() which is the actual immediate source of pain was added in v10, so
it seems like changing the names before that would have more risk than
benefit. Per report from Pavel Raiskup. Discussion:
https://postgr.es/m/4738198.8KVIIDhgEB@nb.usersys.redhat.com
https://git.postgresql.org/pg/commitdiff/003c68a3b45d0d135b874acfe04cf3fb79a6f172

- Disable recheck_on_update optimization to avoid crashes. The code added by
commit c203d6cf8 causes a crash in at least one case, where a
potentially-optimizable expression index has a storage type different from the
input data type. A cursory code review turned up numerous other problems that
seem impractical to fix on short notice. Andres argued for revert of that
patch some time ago, and if additional senior committers had been paying
attention, that's likely what would have happened, but we were not :-( At this
point we can't just revert, at least not in v11, because that would mean an
ABI break for code touching relcache entries. And we should not remove the
(also buggy) support for the recheck_on_update index reloption, since it might
already be used in some databases in the field. So this patch just does the
as-little-invasive-as-possible measure of disabling the feature as though
recheck_on_update were forced off for all indexes. I also removed the related
regression tests (which would otherwise fail) and the user-facing
documentation of the reloption. We should undertake a more thorough code
cleanup if the patch can't be fixed, but not under the extreme time pressure
of being already overdue for 11.1 release. Per report from Ondřej Bouda and
subsequent private discussion among pgsql-release. Discussion:
https://postgr.es/m/20181106185255.776mstcyehnc63ty@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/5d28c9bd73e29890cccd3f6b188b86f81031f671

- Last-minute updates for release notes. Add entries for v11 changes that went
in post-stamping, but before the final wrap.
https://git.postgresql.org/pg/commitdiff/77366d90f44a4e1c15d36e51061ffbc17f0dd12d

- Postpone calculating total_table_pages until after pruning/exclusion. The
planner doesn't make any use of root->total_table_pages until it estimates
costs of indexscans, so we don't need to compute it as early as that's
currently done. By doing the calculation between set_base_rel_sizes and
set_base_rel_pathlists, we can omit relations that get removed from the query
by partition pruning or constraint exclusion, which seems like a more accurate
basis for costing. (Historical note: I think at the time this code was
written, there was not a separation between the "set sizes" and "set
pathlists" steps, so that this approach would have been impossible at the
time. But now that we have that separation, this is clearly the better way to
do things.) David Rowley, reviewed by Edmund Horner Discussion:
https://postgr.es/m/CAKJS1f-NG1mRM0VOtkAG7=ZLQWihoqees9R4ki3CKBB0-fRfCA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/c6e4133fae1fde93769197379ffcc2b379845113

- Fix inadequate autoconfiscation of copyfile() usage. Per buildfarm,
HAVE_COPYFILE is not the same thing as HAVE_COPYFILE_H. Add the extra
configure test.
https://git.postgresql.org/pg/commitdiff/c3e6d5d38699812e047270a711be1926b7f6f396

- Last-minute updates for release notes. Security: CVE-2018-16850
https://git.postgresql.org/pg/commitdiff/1eaeb02a30b307bad4d6a9cc2d596273a5a38a7a

- Disallow setting client_min_messages higher than ERROR. Previously it was
possible to set client_min_messages to FATAL or PANIC, which had the effect of
suppressing transmission of regular ERROR messages to the client. Perhaps
that seemed like a useful option in the past, but the trouble with it is that
it breaks guarantees that are explicitly made in our FE/BE protocol spec about
how a query cycle can end. While libpq and psql manage to cope with the
omission, that's mostly because they are not very bright; client libraries
that have more semantic knowledge are likely to get confused. Notably, pgODBC
doesn't behave very sanely. Let's fix this by getting rid of the ability to
set client_min_messages above ERROR. In HEAD, just remove the FATAL and PANIC
options from the set of allowed enum values for client_min_messages. (This
change also affects trace_recovery_messages, but that's OK since these aren't
useful values for that variable either.) In the back branches, there was
concern that rejecting these values might break applications that are
explicitly setting things that way. I'm pretty skeptical of that argument,
but accommodate it by accepting these values and then internally setting the
variable to ERROR anyway. In all branches, this allows a couple of tiny
simplifications in the logic in elog.c, so do that. Also respond to the point
that was made that client_min_messages has exactly nothing to do with the
server's logging behavior, and therefore does not belong in the "When To Log"
subsection of the documentation. The "Statement Behavior" subsection is a
better match, so move it there. Jonah Harris and Tom Lane Discussion:
https://postgr.es/m/7809.1541521180@sss.pgh.pa.us Discussion:
https://postgr.es/m/15479-ef0f4cc2fd995ca2@postgresql.org
https://git.postgresql.org/pg/commitdiff/3d360e20c9ae48a8139b4dd84f5a8bacc392d8e9

- Fix missing role dependencies for some schema and type ACLs. This patch fixes
several related cases in which pg_shdepend entries were never made, or were
lost, for references to roles appearing in the ACLs of schemas and/or types.
While that did no immediate harm, if a referenced role were later dropped, the
drop would be allowed and would leave a dangling reference in the object's
ACL. That still wasn't a big problem for normal database usage, but it would
cause obscure failures in subsequent dump/reload or pg_upgrade attempts,
taking the form of attempts to grant privileges to all-numeric role names. (I
think I've seen field reports matching that symptom, but can't find any right
now.) Several cases are fixed here: 1. ALTER DOMAIN SET/DROP DEFAULT would
lose the dependencies for any existing ACL entries for the domain. This case
is ancient, dating back as far as we've had pg_shdepend tracking at all. 2.
If a default type privilege applies, CREATE TYPE recorded the ACL properly but
forgot to install dependency entries for it. This dates to the addition of
default privileges for types in 9.2. 3. If a default schema privilege
applies, CREATE SCHEMA recorded the ACL properly but forgot to install
dependency entries for it. This dates to the addition of default privileges
for schemas in v10 (commit ab89e465c). Another somewhat-related problem is
that when creating a relation rowtype or implicit array type, TypeCreate would
apply any available default type privileges to that type, which we don't
really want since such an object isn't supposed to have privileges of its own.
(You can't, for example, drop such privileges once they've been added to an
array type.) ab89e465c is also to blame for a race condition in the regression
tests: privileges.sql transiently installed globally-applicable default
privileges on schemas, which sometimes got absorbed into the ACLs of schemas
created by concurrent test scripts. This should have resulted in failures
when privileges.sql tried to drop the role holding such privileges; but thanks
to the bug fixed here, it instead led to dangling ACLs in the final state of
the regression database. We'd managed not to notice that, but it became
obvious in the wake of commit da906766c, which allowed the race condition to
occur in pg_upgrade tests. To fix, add a function recordDependencyOnNewAcl to
encapsulate what callers of get_user_default_acl need to do; while the
original call sites got that right via ad-hoc code, none of the later-added
ones have. Also change GenerateTypeDependencies to generate these
dependencies, which requires adding the typacl to its parameter list. (That
might be annoying if there are any extensions calling that function directly;
but if there are, they're most likely buggy in the same way as the core
callers were, so they need work anyway.) While I was at it, I changed
GenerateTypeDependencies to accept most of its parameters in the form of a
Form_pg_type pointer, making its parameter list a bit less unwieldy and
mistake-prone. The test race condition is fixed just by wrapping the addition
and removal of default privileges into a single transaction, so that that
state is never visible externally. We might eventually prefer to separate out
tests of default privileges into a script that runs by itself, but that would
be a bigger change and would make the tests run slower overall. Back-patch
relevant parts to all supported branches. Discussion:
https://postgr.es/m/15719.1541725287@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/fa2952d8eb0423a02fde7b36f748f8de11aec8db

- Fix error-cleanup mistakes in exec_stmt_call(). Commit 15c729347 was a couple
bricks shy of a load: we need to ensure that expr->plan gets reset to NULL on
any error exit, if it's not supposed to be saved. Also ensure that the
stmt->target calculation gets redone if needed. The easy way to exhibit a
problem is to set up code that violates the writable-argument restriction and
then execute it twice. But error exits out of, eg, setup_param_list() could
also break it. Make the existing PG_TRY block cover all of that code to be
sure. Per report from Pavel Stehule. Discussion:
https://postgr.es/m/CAFj8pRAeXNTO43W2Y0Cn0YOVFPv1WpYyOqQrrzUiN6s=dn7gCg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/f26c06a4046b62c04ab4a8ef8632a5f705b6dd2d

Thomas Munro pushed:

- Remove dsm_resize() and dsm_remap(). These interfaces were never used in
core, didn't handle failure of posix_fallocate() correctly and weren't
supported on all platforms. We agreed to remove them in 12. Author: Thomas
Munro Reported-by: Andres Freund Discussion:
https://postgr.es/m/CAA4eK1%2B%3DyAFUvpFoHXFi_gm8YqmXN-TtkFH%2BVYjvDLS6-SFq-Q%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3c60d0fa231f7699af5dd83345919d0423305973

- Remove some remaining traces of dsm_resize(). A couple of obsolete comments
and unreachable blocks remained after commit 3c60d0fa. Discussion:
https://postgr.es/m/CAA4eK1%2B%3DyAFUvpFoHXFi_gm8YqmXN-TtkFH%2BVYjvDLS6-SFq-Q%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/9e12fb02b7ec27ff3707a0a7c31f2421ba62c9df

- Provide pg_pread() and pg_pwrite() for random I/O. Forward to POSIX pread()
and pwrite(), or emulate them if unavailable. The emulation is not perfect as
the file position is changed, so we'll put pg_ prefixes on the names to
minimize the risk of confusion in future patches that might inadvertently try
to mix pread() and read() on the same file descriptor. Author: Thomas Munro
Reviewed-by: Tom Lane, Jesper Pedersen Discussion:
https://postgr.es/m/CAEepm=02rapCpPR3ZGF2vW=SBHSdFYO_bz_f-wwWJonmA3APgw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3fd2a7932ef0708dda57369bb20c0499d905cc82

- Use pg_pread() and pg_pwrite() for data files and WAL. Cut down on system
calls by doing random I/O using offset-based OS routines where available.
Remove the code for tracking the 'virtual' seek position. The only reason
left to call FileSeek() was to get the file's size, so provide a new function
FileSize() instead. Author: Oskari Saarenmaa, Thomas Munro Reviewed-by:
Thomas Munro, Jesper Pedersen, Tom Lane, Alvaro Herrera Discussion:
https://postgr.es/m/CAEepm=02rapCpPR3ZGF2vW=SBHSdFYO_bz_f-wwWJonmA3APgw@mail.gmail.com
Discussion:
https://postgr.es/m/b8748d39-0b19-0514-a1b9-4e5a28e6a208%40gmail.com
Discussion: https://postgr.es/m/a86bd200-ebbe-d829-e3ca-0c4474b2fcb7%40ohmu.fi
https://git.postgresql.org/pg/commitdiff/c24dcd0cfd949bdf245814c4c2b3df828ee7db36

- Remove set-but-unused variable. Clean-up for commit c24dcd0c. Reported-by:
Andrew Dunstan Discussion:
https://postgr.es/m/2d52ff4a-5440-f6f1-7806-423b0e6370cb%402ndQuadrant.com
https://git.postgresql.org/pg/commitdiff/c4f0876fb8d8f84fc0a6fb6aee91de120718664c

Bruce Momjian pushed:

- GUC: adjust effective_cache_size SQL descriptions. Follow on patch for commit
3e0f1a4741f564c1a2fa6e944729d6967355d8c7. Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/369ec766-b947-51bd-4dad-6fb9e026439f@2ndquadrant.com
Backpatch-through: 9.4
https://git.postgresql.org/pg/commitdiff/b43df566b372650a9b9e2a0dd9e695c1f16da14f

Andrew Gierth pushed:

- Optimize nested ConvertRowtypeExpr nodes. A ConvertRowtypeExpr is used to
translate a whole-row reference of a child to that of a parent. The planner
produces nested ConvertRowtypeExpr while translating whole-row reference of a
leaf partition in a multi-level partition hierarchy. Executor then translates
the whole-row reference from the leaf partition into all the intermediate
parent's whole-row references before arriving at the final whole-row
reference. It could instead translate the whole-row reference from the leaf
partition directly to the top-most parent's whole-row reference skipping any
intermediate translations. Ashutosh Bapat, with tests by Kyotaro Horiguchi
and some editorialization by me. Reviewed by Andres Freund, Pavel Stehule,
Kyotaro Horiguchi, Dmitry Dolgov, Tom Lane.
https://git.postgresql.org/pg/commitdiff/5613da4cc7910f4b686b818dd1ee68703d62192f

Andres Freund pushed:

- Use installcheck-parallel in pg_upgrade's testsuite. The installcheck run
takes a sizable fraction of test.sh to run. Using a parallel schedule reduces
that noticably. It's possible that we want to backpatch this at some point,
to reduce buildfarm times, but for now lets just see if this upsets the
buildfarm. Author: Andres Freund Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/da906766cd3bef8290db25bd5c8776c03689f3d5

- Build HashState's hashkeys expression with the correct parent. Previously the
expressions were built with the HashJoinState as a parent. That's incorrect.
Currently this does not appear to be harmful, but for the upcoming 'slot
abstraction' work this proves to be problematic, as the underlying slot types
can differ between Hash and HashJoin. It's possible that this already causes
a problem, but I've not been able to come up with a scenario. Therefore don't
backpatch at this point. Author: Andres Freund Discussion:
https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/5f32b29c18195299e90c1fb6c8945e9a46d772d2

- Fixup for b84a6dafbf triggering assert failure in LLVM debug builds. Author:
Andres Freund
https://git.postgresql.org/pg/commitdiff/c14a8ff27e26155345e554a9636bab8236500bb9

- Fix copy-paste error in errhint() introduced in 691d79a07933. Reported-By:
Petr Jelinek Discussion:
https://postgr.es/m/c95a620b-34f0-7930-aeb5-f7ab804f26cb@2ndquadrant.com
Backpatch: 9.4-, like the previous commit
https://git.postgresql.org/pg/commitdiff/a3fb382e9c166e398763e3bbf706b6c92a647fe1

- Move EEOP_*_SYSVAR evaluation out of line. This mainly de-duplicates code. As
evaluating a system variable isn't the hottest path and the current inline
implementation ends up calling out to an external function anyway, this is OK
from a performance POV. The main motivation for de-duplicating is the
upcoming slot abstraction work, after which there's not guaranteed to be a
HeapTuple backing the slot. Author: Andres Freund, Amit Khandekar Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/b84a6dafbf2bb921baee53c0c1aba7719ee38817

- Don't require return slots for nodes without projection. In a lot of nodes
the return slot is not required. That can either be because the node doesn't
do any projection (say an Append node), or because the node does perform
projections but the projection is optimized away because the projection would
yield an identical row. Slots aren't that small, especially for wide rows, so
it's worthwhile to avoid creating them. It's not possible to just skip
creating the slot - it's currently used to determine the tuple descriptor
returned by ExecGetResultType(). So separate the determination of the result
type from the slot creation. The work previously done internally
ExecInitResultTupleSlotTL() can now also be done separately with
ExecInitResultTypeTL() and ExecInitResultSlot(). That way nodes that aren't
guaranteed to need a result slot, can use ExecInitResultTypeTL() to determine
the result type of the node, and ExecAssignScanProjectionInfo() (via
ExecConditionalAssignProjectionInfo()) determines that a result slot is
needed, it is created with ExecInitResultSlot(). Besides the advantage of
avoiding to create slots that then are unused, this is necessary preparation
for later patches around tuple table slot abstraction. In particular
separating the return descriptor and slot is a prerequisite to allow JITing of
tuple deforming with knowledge of the underlying tuple format, and to avoid
unnecessarily creating JITed tuple deforming for virtual slots. This commit
removes a redundant argument from ExecInitResultTupleSlotTL(). While this
commit touches a lot of the relevant lines anyway, it'd normally still not
worthwhile to cause breakage, except that aforementioned later commits will
touch *all* ExecInitResultTupleSlotTL() callers anyway (but fits worse
thematically). Author: Andres Freund Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/1ef6bd2954c4ec63ff8a2c9c4ebc38251d7ef5c5

- docs: Adapt wal_segment_size docs to fc49e24fa69. Before this change the docs
weren't adapted to the fact that wal_segment_size is now measured in bytes,
rather than multiples of wal_block_size. Author: David Steele Discussion:
https://postgr.es/m/68ea97d6-2ed9-f339-e57d-ab3a33caf3b1@pgmasters.net
Backpatch: 11-, like fc49e24fa69 itself.
https://git.postgresql.org/pg/commitdiff/5fc1670bad9bfa138be1c790d529249e0706d38e

- Combine two flag tests in GetSnapshotData(). Previously the code checked
PROC_IN_LOGICAL_DECODING and PROC_IN_VACUUM separately. As the relevant
variable is marked as volatile, the compiler cannot combine the two tests. As
GetSnapshotData() is pretty hot in a number of workloads, it's worthwhile to
fix that. It'd also be a good idea to get rid of the volatiles altogether.
But for one that's a larger patch, and for another, the code after this change
still seems at least as easy to read as before. Author: Andres Freund
Discussion:
https://postgr.es/m/20181005172955.wyjb4fzcdzqtaxjq@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/5fde047f2b41e71706d4ea9e8eb0a401f64192c1

- Remove ineffective check against dropped columns from slot_getattr(). Before
this commit slot_getattr() checked for dropped columns (returning NULL in that
case), but only after checking for previously deformed columns. As
slot_deform_tuple() does not contain such a check, the check in slot_getattr()
would often not have been reached, depending on previous use of the slot.
These days locking and plan invalidation ought to ensure that dropped columns
are not accessed in query plans. Therefore this commit just drops the
insufficient check in slot_getattr(). It's possible that we'll find some
holes againt use of dropped columns, but if so, those need to be addressed
independent of slot_getattr(), as most accesses don't go through that function
anyway. Author: Andres Freund Discussion:
https://postgr.es/m/20181107174403.zai7fedgcjoqx44p@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/c670d0faace6184216c349a4cf830aa415c58068

- Remove volatiles from {procarray,volatile}.c and fix memory ordering issue.
The use of volatiles in procarray.c largely originated from the time when
postgres did not have reliable compiler and memory barriers. That's not the
case anymore, so we can do better. Several of the functions in procarray.c
can be bottlenecks, and removal of volatile yields mildly better code. The
new state, with explicit memory barriers, is also more correct. The previous
use of volatile did not actually deliver sufficient guarantees on weakly
ordered machines, in particular the logic in GetNewTransactionId() does not
look safe. It seems unlikely to be a problem in practice, but worth fixing.
Thomas and I independently wrote a patch for this. Reported-By: Andres Freund
and Thomas Munro Author: Andres Freund, with cherrypicked changes from a patch
by Thomas Munro Discussion:
https://postgr.es/m/20181005172955.wyjb4fzcdzqtaxjq@alap3.anarazel.de
https://postgr.es/m/CAEepm=1nff0x=7i3YQO16jLA2qw-F9O39YmUew4oq-xcBQBs0g@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/450c7defa6d5f3c9fdb6efc92cca106c7b582cb3

Peter Eisentraut pushed:

- pg_upgrade: Allow use of file cloning. Add another transfer mode --clone to
pg_upgrade (besides the existing --link and the default copy), using special
file cloning calls. This makes the file transfer faster and more space
efficient, achieving speed similar to --link mode without the associated
drawbacks. On Linux, file cloning is supported on Btrfs and XFS (if formatted
with reflink support). On macOS, file cloning is supported on APFS.
Reviewed-by: Michael Paquier <michael(at)paquier(dot)xyz>
https://git.postgresql.org/pg/commitdiff/3a769d8239afdc003c91a56d2d8d5adfadacda5d

- Apply RI trigger skipping tests also for DELETE. The tests added in
cfa0f4255bb0f5550d37a01c4d8fe2966d20040c to skip firing an RI trigger if any
old key value is NULL can also be applied for DELETE. This should give a
performance gain in those cases, and it also saves a lot of duplicate code in
the actual RI triggers. (That code was already dead code for the UPDATE
cases.) Reviewed-by: Daniel Gustafsson <daniel(at)yesql(dot)se>
https://git.postgresql.org/pg/commitdiff/69ee2ff9308bbfe0e0a89667555e901fec5e154b

- Remove dead foreign key optimization code. The ri_KeysEqual() calls in the
foreign-key trigger functions to optimize away some updates are useless
because since adfeef55cbcc5dc72a772777f88c1be05a70dfee those triggers are not
enqueued at all. (It's also not useful to keep these checks as some kind of
backstop, since it's also semantically correct to just run the full check even
with equal keys.) Reviewed-by: Daniel Gustafsson <daniel(at)yesql(dot)se>
https://git.postgresql.org/pg/commitdiff/34479d9a36a280afe1387146871d4c1528cf2cbf

Andrew Dunstan pushed:

- Quiet valgrind complaints following pread/pwrite changes. Per complaints from
buildfarm and elsewhere Patch from Jasper Pedersen Discussion:
https://postgr.es/m/0f419c91-49ab-2399-0143-13063bd97c46@redhat.com
https://git.postgresql.org/pg/commitdiff/517b0d0b5f38c92c1807a40d71ca4bf6428ca7d5

- Use parallel installcheck in vcregress.pl's upgrade test. This is to keep the
test in sync with what's done in test.sh, which acquired this change in commit
da906766c.
https://git.postgresql.org/pg/commitdiff/54ad7282fe7ad80a15533546d3b0a2be2e19e3f6

- Adjust valgrind fix in commit 517b0d0b5f. lousyjack still wasn't happy. I
have tested this modification and it worked.
https://git.postgresql.org/pg/commitdiff/12d5f39b15c31c0adc6c7ed06979af340e494fb9

- Disable MSVC warning caused by recent snprintf.c changes. Discussion:
https://postgr.es/m/05f348de-0c79-d88d-69b7-434ef828bd4d@2ndQuadrant.com
https://git.postgresql.org/pg/commitdiff/7f284debaf1933b322ff54a643c6785362a034d3

Álvaro Herrera pushed:

- Revise attribute handling code on partition creation. The original code to
propagate NOT NULL and default expressions specified when creating a partition
was mostly copy-pasted from typed-tables creation, but not being a great match
it contained some duplicity, inefficiency and bugs. This commit fixes the bug
that NOT NULL constraints declared in the parent table would not be honored in
the partition. One reported issue that is not fixed is that a DEFAULT
declared in the child is not used when inserting through the parent. That
would amount to a behavioral change that's better not back-patched. This
rewrite makes the code simpler: 1. instead of checking for duplicate column
names in its own block, reuse the original one that already did that; 2.
instead of concatenating the list of columns from parent and the one declared
in the partition and scanning the result to (incorrectly) propagate defaults
and not-null constraints, just scan the latter searching the former for a
match, and merging sensibly. This works because we know the list in the
parent is already correct and there can only be one parent. This rewrite
makes ColumnDef->is_from_parent unused, so it's removed on branch master; on
released branches, it's kept as an unused field in order not to cause ABI
incompatibilities. This commit also adds a test case for creating partitions
with collations mismatching that on the parent table, something that is
closely related to the code being patched. No code change is introduced
though, since that'd be a behavior change that could break some (broken)
working applications. Amit Langote wrote a less invasive fix for the original
NOT NULL/defaults bug, but while I kept the tests he added, I ended up not
using his original code. Ashutosh Bapat reviewed Amit's fix. Amit reviewed
mine. Author: Álvaro Herrera, Amit Langote Reviewed-by: Ashutosh Bapat, Amit
Langote Reported-by: Jürgen Strobel (bug #15212) Discussion:
https://postgr.es/m/152746742177.1291.9847032632907407358@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/705d433fd5e29ac173ef46a5acd4a2d7336f5573

- Indicate session name in isolationtester notices. When a session under
isolationtester produces printable notices (NOTICE, WARNING) we were just
printing them unadorned, which can be confusing when debugging. Prefix them
with the session name, which makes things clearer. Author: Álvaro Herrera
Reviewed-by: Hari Babu Kommi Discussion:
https://postgr.es/m/20181024213451.75nh3f3dctmcdbfq@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/a28e10e82e540d45e448bc404af76df8a54008f1

== Pending Patches ==

Pavel Stěhule sent in another revision of a patch to implement \dP (show
partition information) in psql.

Dilip Kumar sent in two more revisions of a patch to help with the UNDO system.

Lam Jun Rong sent in two more revisions of a patch to update and improve the
"Getting Started" documentation.

Kuntal Ghosh sent in another revision of a patch to fix shared memory size for
rollback hash table.

Dilip Kumar sent in another revision of a patch to implement UNDO.

John Dent sent in two revisions of a patch to implement delta refreshes for
materialized views using a WHERE clause.

Peter Geoghegan sent in a patch to implement some questionable/tentative ASC
regress output fixes.

Emre Hasegeli and Álvaro Herrera traded patches to quiet Coverity with respect
to the improved geometry types.

Daisuke Higuchi sent in another revision of a patch to add tests for a fixed bug
in ECPG numerics.

Surafel Temesgen sent in two more revisions of a patch to add an option to
pg_dump to output multi-row INSERTs.

Kyotaro HORIGUCHI sent in a patch to fix an infelicity between slot allocation
and replication_reserved_connections.

Pavel Stěhule sent in two revisions of a patch to fix a bug in xmltable xpath
result processing.

Kyotaro HORIGUCHI sent in two revisions of a patch to fix an issue which
manifested as file_fdw using a program could cause an exit code error when using
LIMIT.

Nikita Glukhov sent in two more revisions of a patch to implement JSONPATH.

Daniel Vérité and Michaël Paquier traded patches to add a csv output format to
psql.

Aleksey Kondratov sent in another revision of a patch to fix some issues in
pg_rewind.

Daniel Gustafsson sent in another revision of a patch to support a custom socket
directory during upgrades.

Amit Langote sent in two more revisions of a patch to move the PartitionBoundInfo
creation code to partbounds.c.

Pavel Stěhule sent in another revision of a patch to fix psql's \conninfo and
\connect when using hostaddr.

Edmund Horner sent in a patch to ExecStoreTuple notes from nodeSeqscan.c and
nodeIndexscan.c.

David Rowley sent in two more revisions of a patch to speed up INSERTs and
UPDATEs to partitioned tables

Yoshikazu Imai sent in a patch to rearrange the order in which
inheritance_make_rel_from_joinlist() translates join_list and join_info_list.

David Rowley sent in two more revisions of a patch to better document
PostgreSQL's limits.

Daniel Gustafsson sent in a patch to avoid requesting unused NULLs from
extconfig.

Haribabu Kommi sent in another revision of a patch to implement
pg_stat_statements_reset() to reset specific query/user/db statistics.

Amit Kapila sent in two more revisions of a patch to fix the initialization of
atomic variables introduced by the group clearing mechanism.

David Rowley sent in a patch, for performance demonstration purposes only, which
does unsafe locking reduction for partitioned INSERT/UPDATEs.

Michaël Paquier sent in a patch to add a TAP test checking data consistency on
standby with minRecoveryPoint.

Kyotaro HORIGUCHI sent in two more revisions of a patch to implement the stats
collector in shared memory rather than on disk.

Konstantin Knizhnik sent in two more revisions of a patch to refactor projection
indexes.

Pavel Stěhule sent in a patch to make LEAST and GREATEST take variadic arrays,
as other variadic functions do.

Michaël Paquier sent in a patch to remove an unnecessary
CommandCounterIncrement() call when doing ON COMMIT DELETE ROWS.

Amit Langote sent in another revision of a patch to allow generalized expression
syntax for partition bounds.

Haribabu Kommi sent in another revision of a patch to allow target session attrs
to accept prefer-read option.

Dmitry Dolgov sent in another revision of a patch to implement generic type
subscripting.

Alexander Kuzmenkov sent in a patch to fix a problem that manifested as
uninterruptible long planning of a query with too many WHERE clauses by limiting
the number of clauses the code attempts to identify uniquely.

Daniel Vérité sent in a patch to implement \copyfrom and \copyto in psql.

Amit Langote and David Rowley traded patches to speed up planning with
partitions.

Michaël Paquier sent in a patch to mark flag output from wal_dump with 0x.

James Coleman sent in a patch to ensure that ScalarArrayOpExpr is not NULL.