== PostgreSQL Weekly News - February 25 2018 ==

From: David Fetter <david(at)fetter(dot)org>
To: PostgreSQL Announce <pgsql-announce(at)postgresql(dot)org>
Subject: == PostgreSQL Weekly News - February 25 2018 ==
Date: 2018-02-26 01:21:32
Message-ID: 20180226012132.GA7111@fetter.org
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-announce

== PostgreSQL Weekly News - February 25 2018 ==

pgConf UK 2018 will be held on July 3, 2018 in London. The CfP is open until
March 31, 2018 at http://www.pgconf.uk/papers

== PostgreSQL Product News ==

repmgr 4.0.3, a replication manager for PostgreSQL, released.

== PostgreSQL Jobs for February ==


== PostgreSQL Local ==

PostgreSQL(at)SCaLE is a two day, two track event which takes place on
March 8-9, 2018, at Pasadena Convention Center, as part of SCaLE 16X.

Nordic PGDay 2018 will be held in Oslo, Norway, at the Radisson Blu Hotel
Nydalen, on March 13, 2018. Registration is open and the schedule is posted.

pgDay Paris 2018 will be held in Paris, France at the Espace Saint-Martin, on
March 15 2018. Registration is open.

PGConf APAC 2018 will be held in Singapore March 22-23, 2018.

The German-speaking PostgreSQL Conference 2018 will take place on April 13th,
2018 in Berlin.

PGConfNepal 2018 will be held May 4-5, 2018 at Kathmandu University, Dhulikhel,

PGCon 2018 will take place in Ottawa on May 29 - June 1, 2018.

Swiss PGDay 2018 will take place in Rapperswil (near Zurich) on June 29, 2018.
The CfP is open February 6, 2018 through April 14, 2018, and registration is
open February 6, 2018 through June 28, 2018.

PGConf.Brazil 2018 will take place in São Paulo, Brazil on August 3-4 2018. The
CfP is open until February 28, 2018.

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm EST5EDT. Please send English
language ones to david(at)fetter(dot)org, German language to pwn(at)pgug(dot)de, Italian
language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Peter Eisentraut pushed:

- Message style fix.

- Fix StaticAssertExpr() under C++. The previous code didn't compile, because
static_assert() must end with a semicolon. To fix, wrap it in a block,
similar to the C code.

- Remove redundant function declaration.

- Error message improvement.

- Update gratuitous use of MD5 in documentation. It seems some people are
bothered by the outdated MD5 appearing in example code. So replace it with
more modern alternatives or by a different example function. Reported-by: Jon
Wolski <jonwolski(at)gmail(dot)com>

- Add user-callable SHA-2 functions. Add the user-callable functions sha224,
sha256, sha384, sha512. We already had these in the C code to support SCRAM,
but there was no test coverage outside of the SCRAM tests. Adding these as
user-callable functions allows writing some tests. Also, we have a
user-callable md5 function but no more modern alternative, which led to wide
use of md5 as a general-purpose hash function, which leads to occasional
complaints about using md5. Also mark the existing md5 functions as
leak-proof. Reviewed-by: Michael Paquier <michael(at)paquier(dot)xyz>

- Fix perlcritic warnings.

- Support parameters in CALL. To support parameters in CALL, move the parse
analysis of the procedure and arguments into the global transformation phase,
so that the parser hooks can be applied. And then at execution time pass the
parameters from ProcessUtility on to ExecuteCallStmt.

- Fix filtering of unsupported relations in logical replication. In the
pgoutput plugin, skip changes for relations that are not publishable, per
is_publishable_class(). This concerns in particular materialized views and
information_schema tables. While those relations cannot be part of a
publication, per existing checks, they will be considered by a FOR ALL TABLES
publication. A subscription would not actually apply changes for those
relations, again per existing checks, but trying to match incoming changes to
local tables on the subscriber would lead to errors if no matching local table
exists. Skipping those changes on the publisher avoids sending useless
changes and eliminates the error. Bug: #15044 Reported-by: Chad Trabant
<chad(at)iris(dot)washington(dot)edu> Reviewed-by: Petr Jelinek

- Check error messages in SSL tests. In tests that check whether a connection
fails, also check the error message. That makes sure that the connection was
rejected for the right reason. This discovered that two tests had their
connection failing for the wrong reason. One test failed because pg_hba.conf
was not set up to allow that user, one test failed because the client key file
did not have the right permissions. Fix those tests and add a new one that is
really supposed to check the file permission issue. Reviewed-by: Michael
Paquier <michael(at)paquier(dot)xyz>

- Add current directory to Perl include path. Recent Perl versions don't have
the current directory in the module include path anymore, so we need to add it
here explicitly to make these scripts continue to work.

- Update headers of generated files. The scripts were changed in
c98c35cd084a25c6cf9b08c76de8b89facd75fe7, but the output files were not
updated to reflect the script changes.

- Use croak instead of die in Perl code when appropriate.

Tom Lane pushed:

- Remove redundant initialization of a local variable. In what was doubtless a
typo, commit bf6c614a2 introduced a duplicate initialization of a local
variable. This made Coverity unhappy, as well as pretty much anybody reading
the code. We don't even have a real use for the local variable, so just
remove it.

- Remove bogus "extern" annotations on function definitions. While this is not
illegal C, project style is to put "extern" only on declarations not
definitions. David Rowley Discussion:

- Fix misbehavior of CTE-used-in-a-subplan during EPQ rechecks. An updating
query that reads a CTE within an InitPlan or SubPlan could get incorrect
results if it updates rows that are concurrently being modified. This is
caused by CteScanNext supposing that nothing inside its recursive ExecProcNode
call could change which read pointer is selected in the CTE's shared
tuplestore. While that's normally true because of scoping considerations, it
can break down if an EPQ plan tree gets built during the call, because
EvalPlanQualStart builds execution trees for all subplans whether they're
going to be used during the recheck or not. And it seems like a pretty shaky
assumption anyway, so let's just reselect our own read pointer here. Per bug
#14870 from Andrei Gorita. This has been broken since CTEs were implemented,
so back-patch to all supported branches. Discussion:

- Fix pg_dump's logic for eliding sequence limits that match the defaults. The
previous coding here applied atoi() to strings that could represent values too
large to fit in an int. If the overflowed value happened to match one of the
cases it was looking for, it would drop that limit value from the output,
leading to incorrect restoration of the sequence. Avoid the unsafe behavior,
and also make the logic cleaner by explicitly calculating the default min/max
values for the appropriate kind of sequence. Reported and patched by Alexey
Bashtanov, though I whacked his patch around a bit. Back-patch to v10 where
the faulty logic was added. Discussion:

- Repair pg_upgrade's failure to preserve relfrozenxid for matviews. This
oversight led to data corruption in matviews, manifesting as "could not access
status of transaction" before our most recent releases, and "found xmin from
before relfrozenxid" errors since then. The proximate cause of the problem
seems to have been confusion between the task of preserving dropped-column
status and the task of preserving frozenxid status. Those are required for
distinct sets of relkinds, and the reasoning was entirely undocumented in the
source code. In hopes of forestalling future errors of the same kind, try to
improve the commentary in this area. In passing, also improve the remarkably
unhelpful comments around pg_upgrade's set_frozenxids(). That's not actually
buggy AFAICS, but good luck figuring out what it does from the old comments.
Per report from Claudio Freire. It appears that bug #14852 from Alexey
Ermakov is an earlier report of the same issue, and there may be other cases
that we failed to identify at the time. Patch by me based on analysis by
Andres Freund. The bug dates back to the introduction of matviews, so
back-patch to all supported branches. Discussion:

- Fix planner failures with overlapping mergejoin clauses in an outer join.
Given overlapping or partially redundant join clauses, for example t1 JOIN t2
ON t1.a = t2.x AND t1.b = t2.x the planner's EquivalenceClass machinery will
ordinarily refactor the clauses as "t1.a = t1.b AND t1.a = t2.x", so that join
processing doesn't see multiple references to the same EquivalenceClass in a
list of join equality clauses. However, if the join is outer, it's incorrect
to derive a restriction clause on the outer side from the join conditions, so
the clause refactoring does not happen and we end up with overlapping join
conditions. The code that attempted to deal with such cases had several
subtle bugs, which could result in "left and right pathkeys do not match in
mergejoin" or "outer pathkeys do not match mergeclauses" planner errors, if
the selected join plan type was a mergejoin. (It does not appear that any
actually incorrect plan could have been emitted.) The core of the problem
really was failure to recognize that the outer and inner relations' pathkeys
have different relationships to the mergeclause list. A join's mergeclause
list is constructed by reference to the outer pathkeys, so it will always be
ordered the same as the outer pathkeys, but this cannot be presumed true for
the inner pathkeys. If the inner sides of the mergeclauses contain multiple
references to the same EquivalenceClass ({t2.x} in the above example) then a
simplistic rendering of the required inner sort order is like "ORDER BY t2.x,
t2.x", but the pathkey machinery recognizes that the second sort column is
redundant and throws it away. The mergejoin planning code failed to account
for that behavior properly. One error was to try to generate cut-down
versions of the mergeclause list from cut-down versions of the inner pathkeys
in the same way as the initial construction of the mergeclause list from the
outer pathkeys was done; this could lead to choosing a mergeclause list that
fails to match the outer pathkeys. The other problem was that the pathkey
cross-checking code in create_mergejoin_plan treated the inner and outer
pathkey lists identically, whereas actually the expectations for them must be
different. That led to false "pathkeys do not match" failures in some cases,
and in principle could have led to failure to detect bogus plans in other
cases, though there is no indication that such bogus plans could be generated.
Reported by Alexander Kuzmenkov, who also reviewed this patch. This has been
broken for years (back to around 8.3 according to my testing), so back-patch
to all supported branches. Discussion:

- Allow auto_explain.log_min_duration to go up to INT_MAX. The previous limit
of INT_MAX / 1000 seems to have been cargo-culted in from somewhere else. Or
possibly the value was converted to microseconds at some point; but in all
supported releases, it's just compared to other values, so there's no need for
the restriction. This change raises the effective limit from ~35 minutes to
~24 days, which conceivably is useful to somebody, and anyway it's more
consistent with the range of the core log_min_duration_statement GUC. Per
complaint from Kevin Bloch. Back-patch to all supported releases.

- Fix brown-paper-bag bug in commit 0a459cec96d3856f476c2db298c6b52f592894e8.
RANGE_OFFSET comparisons need to examine the first ORDER BY column, which
isn't necessarily the first column in the incoming tuples. No idea how this
slipped through initial testing. Per bug #15082 from Zhou Digoal.

- First-draft release notes for 10.3.

- Add window RANGE support for float4, float8, numeric. Commit 0a459cec9 left
this for later, but since time's running out, I went ahead and took care of
it. There are more data types that somebody might someday want RANGE support
for, but this is enough to satisfy all expectations of the SQL standard, which
just says that "numeric, datetime, and interval" types should have RANGE

- Fix thinko in in_range_float4_float8. I forgot the coding rule for correct
use of Float8GetDatumFast. Per buildfarm.

- Release notes for 10.3, 9.6.8, 9.5.12, 9.4.17, 9.3.22.

- Un-break parallel pg_upgrade. Commit b3f840120 changed pg_upgrade so that
it'd actually drop and re-create the template1 and postgres databases in the
new cluster. That works fine, serially. With the -j option it's not so fine,
because other per-database jobs might be launched while the template1 database
is dropped. Since they attempt to connect there to start up, kaboom. This is
the cause of the intermittent failures buildfarm member jacana has been
showing for the last month; evidently it is the only BF member configured to
run the pg_upgrade test with parallelism enabled. Fix by processing template1
separately before we get into the parallel sub-job launch loop. (We could
alternatively have made the postgres DB be the special case, but it seems
likely that template1 will contain less stuff and so we lose less parallelism
with this choice.)

Álvaro Herrera pushed:

- Allow UNIQUE indexes on partitioned tables. If we restrict unique constraints
on partitioned tables so that they must always include the partition key, then
our standard approach to unique indexes already works --- each unique key is
forced to exist within a single partition, so enforcing the unique restriction
in each index individually is enough to have it enforced globally. Therefore
we can implement unique indexes on partitions by simply removing a few
restrictions (and adding others.) Discussion:
https://postgr.es/m/20171222212921.hi6hg6pem2w2t36z@alvherre.pgsql Discussion:
Reviewed-by: Simon Riggs, Jesper Pedersen, Peter Eisentraut, Jaime Casanova,
Amit Langote

- Fix expected output.

- Fix crash in pg_replication_slot_advance. We were trying to use a LSN
variable after releasing its containing slot structure. Reported by: tushar
Author: amul sul Reviewed-by: Petr Jelinek, Masahiko Sawada Discussion:

- Adjust ALTER TABLE docs on partitioned constraints. Move the "additional
restrictions" comment to ALTER TABLE ADD CONSTRAINT instead of ADD CONSTRAINT
USING INDEX; and in the latter instead indicate that partitioned tables are
unsupported Noted by David G. Johnston Discussion:

Magnus Hagander pushed:

- Fix typo. Author: Masahiko Sawada

Andres Freund pushed:

- Use platform independent type for TupleTableSlot->tts_off. Previously tts_off
was, for unknown reasons, of type long. For one that's unnecessary as tuples
are restricted in length, for another long would be a bad choice of type even
if that weren't the case, as it's not reliably wider than an int. Also
HeapTupleHeader->t_len is a uint32. This is split off from a larger patch
implementing JITed tuple deforming. Seems like an independent improvement, as
tiny as it is. Author: Andres Freund

- Blindly attempt to adapt sepgsql regression tests. Commit
bf6c614a2f2c58312b3be34a47e7fb7362e07bcb broke the sepgsql test due to a new
invocation of the function access hook during grouping equal initialization.
The new behaviour seems at least as correct as the old one, so try adapt the
tests. As I've no working sepgsql setup here, this is just going from
buildfarm results. Author: Andres Freund Discussion:

Robert Haas pushed:

- Charge cpu_tuple_cost * 0.5 for Append and MergeAppend nodes. Previously,
Append didn't charge anything at all, and MergeAppend charged only
cpu_operator_cost, about half the value used here. This change might make
MergeAppend plans slightly more likely to be chosen than before, since this
commit increases the assumed cost for Append -- with default values -- by
0.005 per tuple but MergeAppend by only 0.0025 per tuple. Since the
comparisons required by MergeAppend are costed separately, it's not clear why
MergeAppend needs to be otherwise more expensive than Append, so hopefully
this is OK. Prior to partition-wise join, it didn't really matter whether or
not an Append node had any cost of its own, because every plan had to use the
same number of Append or MergeAppend nodes and in the same places. Only the
relative cost of Append vs. MergeAppend made a difference. Now, however, it
is possible to avoid some of the Append nodes using a partition-wise join, so
it's worth making an effort. Pending patches for partition-wise aggregate
care too, because an Append of Aggregate nodes will incur the Append overhead
fewer times than an Aggregate over an Append. Although in most cases this
change will favor the use of partition-wise techniques, it does the opposite
when the join cardinality is greater than the sum of the input cardinalities.
Since this situation arises in an existing regression test, I [rhaas] adjusted
it to keep the overall plan shape approximately the same. Jeevan Chalke, per
a suggestion from David Rowley. Reviewed by Ashutosh Bapat. Some changes by
me. The larger patch series of which this patch is a part was also reviewed
and tested by Antonin Houska, Rajkumar Raghuwanshi, David Rowley, Dilip Kumar,
Konstantin Knizhnik, Pascal Legrand, Rafia Sabih, and me. Discussion:

- Try to stabilize EXPLAIN output in partition_check test. Commit
7d8ac9814bc9bb6df2d845dbabed38d7284c7c2c adjusted these tests in the hope of
preserving the plan shape, but I failed to notice that the three partitions
were, on my local machine, choosing two different plan shapes. This is
probably related to the fact that all three tables have exactly the same row
count. Try to improve the situation by making pht1_e about half as large as
the other two. Per Tom Lane and the buildfarm. Discussion:

- Avoid another valgrind complaint about write() of uninitalized bytes. Peter
Geoghegan, per buildfarm member skink and Andres Freund Discussion:

- postgres_fdw: Fix interaction of PHVs with child joins. Commit
f49842d1ee31b976c681322f76025d7732e860f3 introduced the concept of a child
join, but did not update this code accordingly. Ashutosh Bapat, with cosmetic
changes by me Discussion:

- Remove extra word from comment. Etsuro Fujita Discussion:

- Be lazier about partition tuple routing. It's not necessary to fully
initialize the executor data structures for partitions to which no tuples are
ever routed. Consider, for example, an INSERT statement that inserts only one
row: it only cares about the partition to which that one row is routed. The
new function ExecInitPartitionInfo performs the initialization in question
only when a particular partition is about to receive a tuple. This includes
creating, validating, and saving a pointer to the ResultRelInfo, setting up
for speculative insertions, translating WCOs and initializing the resulting
expressions, translating returning lists and building the appropriate
projection information, and setting up a tuple conversion map. One thing
that's not deferred is locking the child partitions; that seems desirable but
would need more thought. Still, testing shows that this makes single-row
inserts significantly faster on a table with many partitions without harming
the bulk-insert case. Amit Langote, reviewed by Etsuro Fujita, with a few
changes by me Discussion:

- Remove extra words. Thomas Munro Discussion:

- Revise API for partition_rbound_cmp/partition_rbound_datum_cmp. Instead of
passing the PartitionKey, pass just the required bits of it. This allows
these functions to be used without needing the PartitionKey to be available,
which is important for several pending patches. Ashutosh Bapat, reviewed by
Amit Langote, with a comment tweak by me. Discussion:

- Revise API for partition bound search functions. Similar to what commit
b0229235564fbe3a9b1cc115ea738a07e274bf30 for a different set of functions,
pass the required bits of the PartitionKey instead of the whole thing. This
allows these functions to be used without needing the PartitionKey to be
available. Amit Langote. The larger patch series of which this patch is a
part has been reviewed and tested by Ashutosh Bapat, David Rowley, Dilip
Kumar, Jesper Pedersen, Rajkumar Raghuwanshi, Beena Emerson, Kyotaro
Horiguchi, Álvaro Herrera, and me, but especially and in great detail by David
Rowley. Discussion:

Noah Misch pushed:

- Synchronize doc/ copies of src/test/examples/. This is mostly cosmetic, but
it might fix build failures, on some platform, when copying from the
documentation. Back-patch to 9.3 (all supported versions).

== Pending Patches ==

David Rowley sent in another revision of a patch to remove [Merge]Append nodes
which contain a single subpath.

Ildus Kurbangaliev sent in another revision of a patch to add a prefix operator
that works with SP-GiST indexes.

Fabien COELHO sent in another revision of a patch to pgbench to add a method for
determining whether a variable exists.

Artur Zakirov sent in a patch to add a snowball dictionary for Nepali.

Amit Langote sent in a patch to adjust partitioned table tests in
insert_conflict.sql and fix ON CONFLICT DO NOTHING with partitioned indexes.

Konstantin Knizhnik sent in another revision of a patch to split lock chains
into pairs in order to avoid O(N^2) behavior.

Laurenz Albe sent in a patch to implement NEXT VALUE FOR sequence.

Anastasia Lubennikova sent in two revisions of a patch to return a heaptuple
from B-Tree index-only scans.

Joe Conway sent in another revision of a patch to add TOAST tables to some more
catalog tables.

Takayuki Tsunakawa sent in two revisions of a patch to fix the documentation of
how to calculate vm.nr_hugepages.

Takayuki Tsunakawa sent in another revision of a patch to zero-fill WAL blocks
on standbys.

Amit Langote sent in five more revisions of a patch to speed up partition

Takayuki Tsunakawa sent in a patch to change the default value of
wal_sync_method to open_datasync on Linux.

Ashutosh Bapat sent in a patch to fix some expression errors with "FOR UPDATE"
using the postgres_fdw with partition-wise join enabled.

Fabien COELHO sent in another revision of a patch to allow pgbench to specify
scale as a size.

Matheus de Oliveira sent in a patch to add support for add support for uuid,
bool, name, bpchar and anyrange to btree_gin.

Haozhou Wang sent in a patch to Add missing type conversion functions for PL/Python

Matheus de Oliveira sent in a patch to add support for add support for uuid,
bool, name, bpchar and anyrange to btree_gin.

Haozhou Wang sent in a patch to add missing type conversion functions for

Matheus de Oliveira sent in a patch to add support for ON UPDATE/DELETE actions

Aleksander Alekseev sent in a patch to add a few suppression rules for Valgrind.

Andrew Dunstan and David Rowley traded patches to speed up the execution of

Tomas Vondra sent in another revision of a patch to add Bloom filters to hash

Peter Eisentraut sent in a patch for PL/pgsql to allow committing inside a
cursor loop in stored procedures.

Andres Freund sent in a patch to refactor TupleTableSlot.

Peter Eisentraut sent in a patch to support parameters in CALL.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous

Daniel Gustafsson sent in a patch to increase the linebuf in the isolation spec

Daniel Gustafsson sent in a patch to allow # comments in the isolation spec SQL

David Rowley sent in a patch to fix an article in the documentation of ALTER

Peter Eisentraut sent in a patch to use file cloning in pg_upgrade and CREATE

Haribabu Kommi sent in another revision of a patch to implement pluggable

Etsuro Fujita sent in another revision of a patch to implement tuple routing for
foreign partitions.

David Rowley sent in another revision of a patch to prune partitions more
efficiently at runtime.

Magnus Hagander sent in three revisions of a patch to make it possible to turn
on checksums while the instance is running.

Nikolay Shaplov sent in a patch to replace StdRdOptions with individual binary
reloptions representation for each relation kind.

Dmitry Dolgov sent in another revision of a patch to implement generic type
subscripting and use same for arrays and JSONB.

Alexander Kuzmenkov sent in another revision of a patch to allow full merge
joins on comparisons other than equality.

Robert Haas sent in another revision of a patch to implement partition-wise

Thomas Munro sent in another revision of a patch to make parallel queries work
under SERIALIZABLE isolation.

David Rowley sent in a patch to generate PartitionClauseInfos for OR clauses,
which should help prune partitions faster at runtime.

Magnus Hagander sent in four more revisions of a patch to allow workers to
override datallowconn.

Peter Geoghegan sent in a patch to fix double free of tuple with grouping

John Naylor sent in another revision of a patch to refactor how bootstrap data
is created and handled.

Thomas Munro sent in a patch to format the isolation tester results more neatly.

David Rowley sent in another revision of a patch to prune partitions at runtime.

Álvaro Herrera sent in a patch to allow FOR EACH ROW triggers on partitioned

Michaël Paquier sent in a patch to monitor xlogreader garbage.

Peter Eisentraut sent in a patch to allow setting an external command for
prompting for SSL passphrases.

Robert Haas sent in a patch to fix some infelicities between parallel append and

David G. Johnston sent in a patch to fix filtering of unsupported relations in

Thomas Munro sent in a patch to update parallel.sgml's treatment of parallel

Tomas Vondra sent in another revision of a patch to implement multivariate
histograms and MCV lists.

Peter Eisentraut sent in another revision of a patch to add a prokind column,
replacing the proisagg and proiswindow columns.

Tomas Vondra sent in another revision of a patch to implement BRIN multi-range

Peter Eisentraut sent in a patch to remove pg_class.relhaspkey.

Peter Eisentraut sent in a patch to handle heap rewrites better in logical

Michail Nikolaev sent in another revision of a patch to add a count for the

Chapman Flack sent in a patch to zero the headers of unused pages after WAL

Yura Sokolov sent in a patch to add a header for customized qsorts.

Browse pgsql-announce by date

  From Date Subject
Next Message Gilles Darold 2018-02-28 18:54:57 pgFormatter v3.0 released
Previous Message Karsten Hilbert 2018-02-25 22:17:34 Re: pg_dumpall SET default_transaction_read_only = off (was Re: == PostgreSQL Weekly News - January 28 2018 ==)