== PostgreSQL Weekly News - December 17 2017 ==

From: David Fetter <david(at)fetter(dot)org>
To: PostgreSQL Announce <pgsql-announce(at)postgresql(dot)org>
Subject: == PostgreSQL Weekly News - December 17 2017 ==
Date: 2017-12-18 00:12:29
Message-ID: 20171218001228.GB30764@fetter.org
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-announce

== PostgreSQL Weekly News - December 17 2017 ==

repmgr 4.0.1, a replication manager for PostgreSQL, released.

Pyrseas 0.8.0, a set of utilities for comparing and synchronizing Postgres
database schemas, released.

== PostgreSQL Product News ==

== PostgreSQL Jobs for December ==


== PostgreSQL Local ==

FOSDEM PGDay 2018, a one day conference held before the main FOSDEM event will
be held in Brussels, Belgium, on Feb 2nd, 2018.

Prague PostgreSQL Developer Day 2018 (P2D2 2018) is a two-day
conference that will be held on February 14-15 2018 in Prague, Czech Republic.
The CfP is open until January 5, 2018 at https://p2d2.cz/callforpapers

PGConf India 2018 will be on February 22-23, 2018 in Bengaluru, Karnataka.

PostgreSQL(at)SCaLE is a two day, two track event which takes place on
March 8-9, 2018, at Pasadena Convention Center, as part of SCaLE 16X.

Nordic PGDay 2018 will be held in Oslo, Norway, at the Radisson Blu Hotel
Nydalen, on March 13, 2018. The CfP is open through December 31, 2017 at

pgDay Paris 2018 will be held in Paris, France at the Espace Saint-Martin, on
March 15 2018. The CfP is open until December 31, 2017.

PGConf APAC 2018 will be held in Singapore March 22-24, 2018.

The German-speaking PostgreSQL Conference 2018 will take place on April 13th,
2018 in Berlin. The CfP is open until January 09, 2018 at
http://2018.pgconf.de/de/callforpapers.html and the conference site is at

PGCon 2018 will take place in Ottawa on May 29 - June 1, 2018. The CFP is open
until January 19, 2018 at https://www.pgcon.org/2018/papers.php

PGConf.Brazil 2018 will take place in São Paulo, Brazil on August 3-4 2018. The
CfP will open soon.

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm EST5EDT. Please send English
language ones to david(at)fetter(dot)org, German language to pwn(at)pgug(dot)de, Italian
language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Stabilize output of new regression test case. The test added by commit
390d58135 turns out to have different output in CLOBBER_CACHE_ALWAYS builds:
there's an extra CONTEXT line in the error message as a result of detecting
the error at a different place. Possibly we should do something to make that
more consistent. But as a stopgap measure to make the buildfarm green again,
adjust the test to suppress CONTEXT entirely. We can revert this if we do
something in the backend to eliminate the inconsistency. Discussion:

- Fix corner-case coredump in _SPI_error_callback(). I noticed that
_SPI_execute_plan initially sets spierrcontext.arg = NULL, and only fills it
in some time later. If an error were to happen in between,
_SPI_error_callback would try to dereference the null pointer. This is
unlikely --- there's not much between those points except push-snapshot calls
--- but it's clearly not impossible. Tweak the callback to do nothing if the
pointer isn't set yet. It's been like this for awhile, so back-patch to all
supported branches.

- Rethink MemoryContext creation to improve performance. This patch makes a
number of interrelated changes to reduce the overhead involved in
creating/deleting memory contexts. The key ideas are: * Include the
AllocSetContext header of an aset.c context in its first malloc request,
rather than allocating it separately in TopMemoryContext. This means that we
now always create an initial or "keeper" block in an aset, even if it never
receives any allocation requests. * Create freelists in which we can save and
recycle recently-destroyed asets (this idea is due to Robert Haas). * In the
common case where the name of a context is a constant string, just store a
pointer to it in the context header, rather than copying the string. The
first change eliminates a palloc/pfree cycle per context, and also avoids
bloat in TopMemoryContext, at the price that creating a context now involves a
malloc/free cycle even if the context never receives any allocations. That
would be a loser for some common usage patterns, but recycling short-lived
contexts via the freelist eliminates that pain. Avoiding copying constant
strings not only saves strlen() and strcpy() overhead, but is an essential
part of the freelist optimization because it makes the context header size
constant. Currently we make no attempt to use the freelist for contexts with
non-constant names. (Perhaps someday we'll need to think harder about that,
but in current usage, most contexts with custom names are long-lived anyway.)
The freelist management in this initial commit is pretty simplistic, and we
might want to refine it later --- but in common workloads that will never
matter because the freelists will never get full anyway. To create a context
with a non-constant name, one is now required to call
AllocSetContextCreateExtended and specify the MEMCONTEXT_COPY_NAME option.
AllocSetContextCreate becomes a wrapper macro, and it includes a test that
will complain about non-string-literal context name parameters on gcc and
similar compilers. An unfortunate side effect of making AllocSetContextCreate
a macro is that one is now *required* to use the size parameter abstraction
macros (ALLOCSET_DEFAULT_SIZES and friends) with it; the pre-9.6 habit of
writing out individual size parameters no longer works unless you switch to
AllocSetContextCreateExtended. Internally to the memory-context-related
modules, the context creation APIs are simplified, removing the rather baroque
original design whereby a context-type module called mcxt.c which then called
back into the context-type module. That saved a bit of code duplication, but
not much, and it prevented context-type modules from exercising control over
the allocation of context headers. In passing, I converted the test-and-elog
validation of aset size parameters into Asserts to save a few more cycles.
The original thought was that callers might compute size parameters on the
fly, but in practice nobody does that, so it's useless to expend cycles on
checking those numbers in production builds. Also, mark the memory context
method-pointer structs "const", just for cleanliness. Discussion:

- Tighten configure's test for __builtin_constant_p(). Commit 9fa6f00b1 assumed
that __builtin_constant_p("string literal") is TRUE, if the compiler has that
function at all. Buildfarm results show that Sun Studio 12, at least, breaks
that assumption. Removing that usage would leave us with no mechanical check
for a very fragile coding requirement, so instead teach configure to ignore
__builtin_constant_p() if it doesn't behave that way. We could complicate
matters by distinguishing three cases (no such function, vs does, vs doesn't
work for string literals); but for now, that seems unnecessary because our
other existing uses of this function are just fairly minor optimizations of
non-returning elog/ereport. We can live without that on the small population
of compilers that act this way. Discussion:

- Fix oversights in new plpgsql test suite infrastructure. Fix a couple of
minor oversights in commit 632b03da3: the tests should be run in database
"pl_regression" like the other PLs do, and we should clean up the tests'
output cruft during "make clean".

- Suppress compiler warning about no function return value. Compilers that
don't know that ereport(ERROR) doesn't return complained about the new coding
in scanint8() introduced by commit 101c7ee3e. Tweak coding to avoid the
warning. Per buildfarm.

- Try harder to detect unavailability of __builtin_mul_overflow(int64). Commit
c04d35f44 didn't quite do the job here, because it still allowed the compiler
to deduce that the function call could be optimized away. Prevent that by
putting the arguments and results in global variables. Discussion:

Robert Haas pushed:

- Improve comment about PartitionBoundInfoData. Ashutosh Bapat, per discussion
with Julien Rouhaund, who also reviewed this patch. Discussion:

- Remove bug from OPTIMIZER_DEBUG code for partition-wise join. Etsuro Fujita,
reviewed by Ashutosh Bapat Discussion:

- Remove obsolete comment. Commit 8b304b8b72b0a60f1968d39f01cf817c8df863ec
removed replacement selection, but left behind this comment text. The
optimization to which the comment refers is not relevant without replacement
selection, because if we had so few tuples as to require only one tape, we
would have just completed the sort in memory. Peter Geoghegan Discussion:

- Revert "Fix accumulation of parallel worker instrumentation.". This reverts
commit 2c09a5c12a66087218c7f8cba269cd3de51b9b82. Per further discussion, that
doesn't seem to be the best possible fix. Discussion:

- Fix parallel index scan hang with deleted or half-dead pages. The previous
coding forgot to release the scan before seizing it again, leading to a
lockup. Report by Patrick Hemmer. Diagnosis by Thomas Munro. Patch by Amit
Kapila. Discussion:

Peter Eisentraut pushed:

- Fix comment. Reported-by: Noah Misch <noah(at)leadboat(dot)com>

- PL/Python: Fix potential NULL pointer dereference. After
d0aa965c0a0ac2ff7906ae1b1dad50a7952efa56, one error path in
PLy_spi_execute_fetch_result() could result in the variable "result" being
dereferenced after being set to NULL. Rearrange the code a bit to fix that.
Also add another SPI_freetuptable() call so that that is cleared in all error
paths. discovered by John Naylor <jcnaylor(at)gmail(dot)com> via scan-build ideas
and review by Tom Lane

- Fix crash when using CALL on an aggregate. Author: Ashutosh Bapat
<ashutosh(dot)bapat(at)enterprisedb(dot)com> Reported-by: Rushabh Lathia

- Start a separate test suite for plpgsql. The plpgsql.sql test file in the
main regression tests is now by far the largest after numeric_big, making
editing and managing the test cases very cumbersome. The other PLs have their
own test suites split up into smaller files by topic. It would be nice to
have that for plpgsql as well. So, to get that started, set up test
infrastructure in src/pl/plpgsql/src/ and split out the recently added
procedure test cases into a new file there. That file now mirrors the test
cases added to the other PLs, making managing those matching tests a bit
easier too. msvc build system changes with help from Michael Paquier

Teodor Sigaev pushed:

- Make pg_trgm tests independ from standard_conforming_string. Tests uses.
regular expression which contains backslash.

- Add approximated Zipfian-distributed random generator to pgbench. Generator
helps to make close to real-world tests. Author: Alik Khilazhev Reviewed-By:
Fabien COELHO Discussion:

Andres Freund pushed:

- Make PGAC_C_BUILTIN_OP_OVERFLOW link instead of just compiling. Otherwise the
detection can spuriously detect symbol as available, because the compiler may
just emits reference to non-existant symbol.

- Consistently use PG_INT(16|32|64)_(MIN|MAX). Per buildfarm animal woodlouse.

- Add float.h include to int8.c, for isnan(). port.h redirects isnan() to
_isnan() on windows, which in turn is provided by float.h rather than math.h.
Therefore include the latter as well. Per buildfarm.

- Add defenses against pre-crash files to BufFileOpenShared(). Crash restarts
currently don't clean up temporary files, as a debugging aid. If a left-over
file happens to have the same name as a segment file we're trying to create,
we'll just truncate and reuse it, but there is a problem: BufFileOpenShared()
determines how many segment files exist by trying to open .0, .1, .2, ...
until it finds no more files. It might be confused by a junk file that has
the next segment number. To defend against that, make sure we always create a
gap after the end file by unlinking the following name if it exists. Also
make it an error to try to open a BufFile that doesn't exist (has no segment
0), so as not to encourage the development of client code that depends on an
interface that we can't reliably provide. Author: Thomas Munro Reviewed-By:
Andres Freund Discussion:

- Add pg_attribute_always_inline. Sometimes it is useful to be able to insist
that the compiler inline a function that its normal cost analysis would not
normally choose to inline. This can be useful for instantiating different
variants of a function that remove branches of code by constant folding.
Author: Thomas Munro Reviewed-By: Andres Freund Discussion:

- Allow executor nodes to change their ExecProcNode function. In order for
executor nodes to be able to change their ExecProcNode function after
ExecInitNode() has finished, provide ExecSetExecProcNode(). This allows any
wrappers functions that only execProcnode.c knows about to be reinstalled.
The motivation for wanting to change ExecProcNode after ExecInitNode() has
finished is that it is not known until later whether parallel query is
available, so if a parallel variant is to be installed then ExecInitNode() is
too soon to decide. Author: Thomas Munro Reviewed-By: Andres Freund

- Fix a number of copy & paste comment errors in common/int.h. Author:
Christoph Berg Discussion:

- Fix pruning of locked and updated tuples. Previously it was possible that a
tuple was not pruned during vacuum, even though its update xmax (i.e. the
updating xid in a multixact with both key share lockers and an updater) was
below the cutoff horizon. As the freezing code assumed, rightly so, that
that's not supposed to happen, xmax would be preserved (as a member of a new
multixact or xmax directly). That causes two problems: For one the tuple is
below the xmin horizon, which can cause problems if the clog is truncated or
once there's an xid wraparound. The bigger problem is that that will break HOT
chains, which in turn can lead two to breakages: First, failing index lookups,
which in turn can e.g lead to constraints being violated. Second, future hot
prunes / vacuums can end up making invisible tuples visible again. There's
other harmful scenarios. Fix the problem by recognizing that tuples can be
DEAD instead of RECENTLY_DEAD, even if the multixactid has alive members, if
the update_xid is below the xmin horizon. That's safe because newer versions
of the tuple will contain the locking xids. A followup commit will harden the
code somewhat against future similar bugs and already corrupted data. Author:
Andres Freund, with changes by Alvaro Herrera Reported-By: Daniel Wood
Analyzed-By: Andres Freund, Alvaro Herrera, Robert Haas, Peter Geoghegan,
Daniel Wood, Yi Wen Wong, Michael Paquier Reviewed-By: Alvaro Herrera, Robert
Haas, Michael Paquier Discussion:
Backpatch: 9.3-

- Perform a lot more sanity checks when freezing tuples. The previous commit
has shown that the sanity checks around freezing aren't strong enough.
Strengthening them seems especially important because the existance of the bug
has caused corruption that we don't want to make even worse during future
vacuum cycles. The errors are emitted with ereport rather than elog, despite
being "should never happen" messages, so a proper error code is emitted. To
avoid superflous translations, mark messages as internal. Author: Andres
Freund and Alvaro Herrera Reviewed-By: Alvaro Herrera, Michael Paquier
Backpatch: 9.3-

- Try to detect runtime unavailability of __builtin_mul_overflow(int64). On
some systems the results of 64 bit __builtin_mul_overflow() operations can be
computed at compile time, but not at runtime. The known cases are arm buildfar
animals using clang where the runtime operation is implemented in a
unavailable function. Try to avoid compile-time computation by using volatile
arguments to __builtin_mul_overflow(). In that case we hopefully will get a
link error when unavailable, similar to what buildfarm animals dangomushi and
gull are reporting. Author: Andres Freund Discussion:

- Provide overflow safe integer math inline functions. It's not easy to get
signed integer overflow checks correct and fast. Therefore abstract the
necessary infrastructure into a common header providing addition, subtraction
and multiplication for 16, 32, 64 bit signed integers. The new macros aren't
yet used, but a followup commit will convert several open coded overflow
checks. Author: Andres Freund, with some code stolen from Greg Stark
Reviewed-By: Robert Haas Discussion:

- Use new overflow aware integer operations. A previous commit added inline
functions that provide fast(er) and correct overflow checks for signed integer
math. Use them in a significant portion of backend code. There's more to
touch in both backend and frontend code, but these were the easily
identifiable cases. The old overflow checks are noticeable in integer heavy
workloads. A secondary benefit is that getting rid of overflow checks that
rely on signed integer overflow wrapping around, will allow us to get rid of
-fwrapv in the future. Which in turn slows down other code. Author: Andres
Freund Discussion:

Andrew Dunstan pushed:

- Fix walsender timeouts when decoding a large transaction. The logical slots
have a fast code path for sending data so as not to impose too high a per
message overhead. The fast path skips checks for interrupts and timeouts.
However, the existing coding failed to consider the fact that a transaction
with a large number of changes may take a very long time to be processed and
sent to the client. This causes the walsender to ignore interrupts for
potentially a long time and more importantly it will result in the walsender
being killed due to timeout at the end of such a transaction. This commit
changes the fast path to also check for interrupts and only allows calling the
fast path when the last keepalive check happened less than half the walsender
timeout ago. Otherwise the slower code path will be taken. Backpatched to 9.4
Petr Jelinek, reviewed by Kyotaro HORIGUCHI, Yura Sokolov, Craig Ringer and
Robert Haas. Discussion:

Noah Misch pushed:

- Avoid and detect SIGPIPE race in TAP tests. Don't write to stdin of a psql
process that could have already exited with an authentication failure.
Buildfarm members crake and mandrill have failed once by doing so. Ignore
SIGPIPE in all TAP tests. Back-patch to v10, where these tests were
introduced. Reviewed by Michael Paquier. Discussion:

== Pending Patches ==

Amit Langote sent in a patch to error out if the left hand side of ANY/ALL is
not a scalar.

Etsuro Fujita sent in another revision of a patch to fix a bug in the PostgreSQL

Kyotaro HORIGUCHI sent in another revision of a patch to fix an issue with WAL.

Kyotaro HORIGUCHI sent in another revision of a patch to implement asynchronous
execution infrastructure and use same in the PostgreSQL FDW.

Kyotaro HORIGUCHI sent in another revision of a patch to collect more stats
about skipped vacuums.

Ashutosh Bapat sent in a patch to fix a comment in partition.c.

Alexander Korotkov sent in two more revisions of a patch to fix some
infelicities in the the trigram supplied extension with respect to word

Amit Kapila and Robert Haas traded patches to fix parallel.c obliviousness
about worker startup failures.

Masahiko Sawada and Fujii Masao traded patches to fix an assertion failure when
the non-exclusive pg_stop_backup aborted.

Masahiko Sawada and Robert Haas traded patches to move relation extension locks
out of heavyweight lock manager.

Haribabu Kommi sent in another revision of a patch to implement the
infrastructure for pluggable storage.

Kyotaro HORIGUCHI sent in another revision of a patch to fix a race between

Rushabh Lathia sent in another revision of a patch to implement parallel
tuplesort for index creation.

Etsuro Fujita sent in another revision of a patch to implement tuple routing to
foreign partitions.

Nikhil Sontakke sent in another revision of a patch to implement logical
decoding of two-phase transactions.

Haribabu Kommi sent in another revision of a patch to implement

Amit Langote sent in three revisions of a patch to allow boolean values in
partition FOR VALUES clause.

Amit Langote sent in another revision of a patch to make partition pruning

Thomas Munro sent in two more revisions of a patch to implement parallel hash.

David Steele sent in a patch to exclude unlogged tables from base backups.

Peter Eisentraut sent in two revisions of a patch to use portal pinning in
PL/Perl and PL/PythonU.

Peter Eisentraut sent in another revision of a patch to replace GrantObjectType
with ObjectType.

Beena Emerson and David Rowley traded patches to do runtime partition pruning.

Bill Moyers sent in a patch to fix a possible NULL dereference in str_time.

Haribabu Kommi sent in another revision of a patch to refactor handling of
database attributes between pg_dump and pg_dumpall.

Amit Langote sent in a patch to emit list partition constraint as OR expression,
making it round-trippable.

Andres Freund sent in a patch to remove out of date types.

John Naylor sent in two more revisions of a patch to make bootstrap data easier
to use.

Ali Akbar sent in two revisions of a patch to add SET NOT NULL in inheritance
children when needed.

Amit Khandekar sent in another revision of a patch to enable UPDATEs to
partitioned keys which would have the effect of moving tuples to other

Fabien COELHO sent in another revision of a patch to add \if to pgbench.

Ali Akbar sent in two revisions of a patch to pg_upgrade to ensure that NOT NULL
in inheritance children is consistent.

Michaël Paquier sent in a patch to fix some infelicities in the overflow-safe
integer math inline function on ARM.

Chapman Flack sent in a patch to give worker_spi.naptime an explicit time unit.

Konstantin Knizhnik sent in two more revisions of a patch to add a
recheck_on_update property to indexes, signaling that they are non-injective.

Fabien COELHO sent in three more revisions of a patch to add more functions and
operators to pgbench.

Amit Langote sent in a patch to teach dbsize.c functions about partitioned

Chapman Flack sent in a patch to clarify that a BGW can register a dynamic BGW.

Kyotaro HORIGUCHI sent in a patch to fix a bug where autoprewarm is fogetting to
register a tranche.

Maksim Milyutin sent in a patch to fix a wrong t_bits alignment in pageinspect.

Peter Geoghegan sent in a patch to promote "HOT parent tuple" elog to an

Christoph Berg sent in a patch to ensure that genbki.pl always produces
reproducible output.

Christoph Berg sent in a patch to add support for VENDOR_VERSION to configure.

Thomas Munro sent in a patch to fix an infelicity between top-N sorts and
parallel execution.

David Rowley sent in a patch to fix a typo in a comment in json_agg_transfn.

David Rowley sent in a patch to parallelize string_agg and array_agg.

Browse pgsql-announce by date

  From Date Subject
Next Message Peter Eisentraut 2017-12-19 16:31:11 PgBouncer 1.8 released
Previous Message Monica Real Amores 2017-12-13 08:25:18 repmgr 4.0.1 Now Available