== PostgreSQL Weekly News - September 23, 2018 ==

From: David Fetter <david(at)fetter(dot)org>
To: PostgreSQL Announce <pgsql-announce(at)postgresql(dot)org>
Subject: == PostgreSQL Weekly News - September 23, 2018 ==
Date: 2018-09-23 19:14:23
Message-ID: 20180923191423.GA27799@fetter.org
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-announce

== PostgreSQL Weekly News - September 23, 2018 ==

The PostgreSQL project's Code of Conduct (CoC) has now been
published. Please take some time to read and understand it.

PostgreSQL 11 Beta 4 released. Test!

== PostgreSQL Product News ==

Pgpool-II 4.0 beta1 released.

pitrery 2.2, a set of Bash scripts to manage PITR backups for PostgreSQL, released.

== PostgreSQL Jobs for September ==


== PostgreSQL Local ==

PostgresConf South Africa 2018 will take place in Johannesburg on October 9, 2018

PostgreSQL Conference Europe 2018 will be held on October 23-26, 2018 at the
Lisbon Marriott Hotel in Lisbon, Portugal.

2Q PGConf will be on December 4-5, 2018 in Chicago, IL.

PGConf.ASIA 2018 will take place on December 10-12, 2018 in Akihabara, Tokyo,

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

- Refactor routines for subscription and publication lookups. Those routines
gain a missing_ok argument, allowing a caller to get a NULL result instead of
an error if set to true. This is part of a larger refactoring effort for
objectaddress.c where trying to check for non-existing objects does not result
in cache lookup failures. Author: Michael Paquier Reviewed-by: Aleksander
Alekseev, Álvaro Herrera Discussion:

- Enforce translation mode for Windows frontends to text with open/fopen.
Allowing frontends to use concurrent-safe open() and fopen() via 0ba06e0 has
the side-effect of switching the default translation mode from text to binary,
so the switch can cause breakages for frontend tools when the caller of those
new versions specifies neither binary and text. This commit makes sure to
maintain strict compatibility with past versions, so as no frontends should
see a difference when upgrading. Author: Laurenz Albe Reviewed-by: Michael
Paquier, Tom Lane Discussion:

- Remove special handling for open() in initdb for Windows. 40cfe86 enforces
the translation mode to text for all frontends, so this special handling in
initdb is not needed anymore.

- Document lock taken on referenced table when adding a foreign key. This can
happen for CREATE TABLE and ALTER TABLE, so a mention is added to both of them
in the concerned subsections. Author: Adrien Nayrat Discussion:

- Make GUC wal_sender_timeout user-settable. Being able to use a value that can
be changed on a connection basis is useful with clusters distributed
geographically, and makes failure detection more flexible. A note is added in
the documentation about the use of "options" in primary_conninfo, which can be
hard to grasp for newcomers with the need of two single quotes when listing a
set of parameters. Author: Tsunakawa Takayuki Reviewed-by: Masahiko Sawada,
Michael Paquier Discussion:

Thomas Munro pushed:

- Allow DSM allocation to be interrupted. Chris Travers reported that the
startup process can repeatedly try to cancel a backend that is in a
posix_fallocate()/EINTR loop and cause it to loop forever. Teach the retry
loop to give up if an interrupt is pending. Don't actually check for
interrupts in that loop though, because a non-local exit would skip some
clean-up code in the caller. Back-patch to 9.4 where DSM was added (and
posix_fallocate() was later back-patched). Author: Chris Travers Reviewed-by:
Ildar Musin, Murat Kabilov, Oleksii Kliukin Tested-by: Oleksii Kliukin

- Defer restoration of libraries in parallel workers. Several users of
extensions complained of crashes in parallel workers that turned out to be due
to syscache access from their _PG_init() functions. Reorder the
initialization of parallel workers so that libraries are restored after the
caches are initialized, and inside a transaction. This was reported in bug
#15350 and elsewhere. We don't consider it to be a bug: extensions shouldn't
do that, because then they can't be used in shared_preload_libraries.
However, it's a fairly obscure hazard and these extensions worked in practice
before parallel query came along. So let's make it work. Later commits might
add a warning message and eventually an error. Back-patch to 9.6, where
parallel query landed. Author: Thomas Munro Reviewed-by: Amit Kapila
Reported-by: Kieran McCusker, Jimmy Discussion:

- Fix segment_bins corruption in dsa.c. If a segment has been freed by dsa.c
because it is entirely empty, other backends must make sure to unmap it before
following links to new segments that might happen to have the same index
number, or they could finish up looking at a defunct segment and then corrupt
the segment_bins lists. The correct protocol requires checking
freed_segment_counter after acquiring the area lock and before resolving any
index number to a segment. Add the missing checks and an assertion.
Back-patch to 10, where dsa.c first arrived. Author: Thomas Munro
Reported-by: Tomas Vondra Discussion:

- Use size_t consistently in dsa.{ch}. Takeshi Ideriha complained that there is
a mixture of Size and size_t in dsa.c and corresponding header. Let's use
size_t. Back-patch to 10 where dsa.c landed, to make future back-patching
easy. Discussion:

Tom Lane pushed:

- Fix some probably-minor oversights in readfuncs.c. The system expects
TABLEFUNC RTEs to have coltypes, coltypmods, and colcollations lists, but
outfuncs doesn't dump them and readfuncs doesn't restore them. This doesn't
cause obvious failures, because the only things that look at those fields are
expandRTE() and get_rte_attribute_type(), which are mostly used during parse
analysis, before anything would've passed the parsetree through
outfuncs/readfuncs. But expandRTE() is used in build_physical_tlist(), which
means that that function will return a wrong answer for a TABLEFUNC RTE that
came from a view. Very accidentally, this doesn't cause serious problems,
because what it will return is NIL which callers will interpret as "couldn't
build a physical tlist because of dropped columns". So you still get a plan
that works, though it's marginally less efficient than it could be. There are
also some other expandRTE() calls associated with transformation of whole-row
Vars in the planner. I have been unable to exhibit misbehavior from that, and
it may be unreachable in any case that anyone would care about ... but I'm not
entirely convinced, so this seems like something we should back- patch a fix
for. Fortunately, we can fix it without forcing a change of stored rules and
a catversion bump, because we can just copy these lists from the subsidiary
TableFunc object. readfuncs.c was also missing support for
NamedTuplestoreScan plan nodes. This accidentally fails to break parallel
query because a query using a named tuplestore would never be considered
parallel-safe anyway. However, project policy since parallel query came in is
that all plan node types should have outfuncs/readfuncs support, so this is
clearly an oversight that should be repaired. Noted while fooling around with
a patch to test outfuncs/readfuncs more thoroughly. That exposed some other
issues too, but these are the only ones that seem worth back-patching.
Back-patch to v10 where both of these features came in. Discussion:

- Fix some minor issues exposed by outfuncs/readfuncs testing. A test patch to
pass parse and plan trees through outfuncs + readfuncs exposed several issues
that need to be fixed to get clean matches: Query.withCheckOptions failed to
get copied; it's intentionally ignored by outfuncs/readfuncs on the grounds
that it'd always be NIL anyway in stored rules. This seems less than
future-proof, and it's not even saving very much, so just undo the decision
and treat the field like all others. Several places that convert a view RTE
into a subquery RTE, or similar manipulations, failed to clear out fields that
were specific to the original RTE type and should be zero in a subquery RTE.
Since readfuncs.c will leave such fields as zero, equalfuncs.c thinks the
nodes are different leading to a reported mismatch. It seems like a good idea
to clear out the no-longer-needed fields, even though in principle nothing
should look at them; the node ought to be indistinguishable from how it would
look if we'd built a new node instead of scribbling on the old one.
BuildOnConflictExcludedTargetlist randomly set the resname of some
TargetEntries to "" not NULL. outfuncs/readfuncs don't distinguish those
cases, and so the string will read back in as NULL ... but equalfuncs.c does
distinguish. Perhaps we ought to try to make things more consistent in this
area --- but it's just useless extra code space for
BuildOnConflictExcludedTargetlist to not use NULL here, so I fixed it for now
by making it do that. catversion bumped because the change in handling of
Query.withCheckOptions affects stored rules. Discussion:

- Add a debugging option to stress-test outfuncs.c and readfuncs.c. In the
normal course of operation, query trees will be serialized only if they are
stored as views or rules; and plan trees will be serialized only if they get
passed to parallel-query workers. This leaves an awful lot of opportunity for
bugs/oversights to not get detected, as indeed we've just been reminded of the
hard way. To improve matters, this patch adds a new compile option
WRITE_READ_PARSE_PLAN_TREES, which is modeled on the longstanding option
COPY_PARSE_PLAN_TREES; but instead of passing all parse and plan trees through
copyObject, it passes them through nodeToString + stringToNode. Enabling this
option in a buildfarm animal or two will catch problems at least for cases
that are exercised by the regression tests. A small problem with this idea is
that readfuncs.c historically has discarded location fields, on the reasonable
grounds that parse locations in a retrieved view are not relevant to the
current query. But doing that in WRITE_READ_PARSE_PLAN_TREES breaks
pg_stat_statements, and it could cause problems for future improvements that
might try to report error locations at runtime. To fix that, provide a
variant behavior in readfuncs.c that makes it restore location fields when
told to. In passing, const-ify the string arguments of stringToNode and its
subsidiary functions, just because it annoyed me that they weren't const
already. Discussion: https://postgr.es/m/17114.1537138992@sss.pgh.pa.us

- Don't ignore locktable-full failures in StandbyAcquireAccessExclusiveLock.
Commit 37c54863c removed the code in StandbyAcquireAccessExclusiveLock that
checked the return value of LockAcquireExtended. That created a bug, because
it's still passing reportMemoryError = false to LockAcquireExtended, meaning
that LOCKACQUIRE_NOT_AVAIL will be returned if we're out of shared memory for
the lock table. In such a situation, the startup process would believe it had
acquired an exclusive lock even though it hadn't, with potentially dire
consequences. To fix, just drop the use of reportMemoryError = false, which
allows us to simplify the call into a plain LockAcquire(). It's unclear that
the locktable-full situation arises often enough that it's worth having a
better recovery method than crash-and-restart. (I strongly suspect that the
only reason the code path existed at all was that it was relatively simple to
do in the pre-37c54863c implementation. But now it's not.)
LockAcquireExtended's reportMemoryError parameter is now dead code and could
be removed. I refrained from doing so, however, because there was some
interest in resurrecting the behavior if we do get reports of locktable-full
failures in the field. Also, it seems unwise to remove the parameter
concurrently with shipping commit f868a8143, which added a parameter; if there
are any third-party callers of LockAcquireExtended, we want them to get a
wrong-number-of-parameters compile error rather than a possibly-silent
misinterpretation of its last parameter. Back-patch to 9.6 where the bug was
introduced. Discussion: https://postgr.es/m/6202.1536359835@sss.pgh.pa.us

- Fix minor error message style guide violation. No periods at the ends of
primary error messages, please. Daniel Gustafsson Discussion:

- Teach genbki.pl to auto-generate pg_type entries for array types. This
eliminates some more tedium in adding new catalog entries, specifically the
need to set up an array type when adding a new built-in data type. Now it's
sufficient to assign an OID for the array type and write it in an
"array_type_oid" metadata field. You don't have to fill the base type's
typarray link explicitly, either. No catversion bump since the contents of
pg_type aren't changed. (Well, their order might be different, but that
doesn't matter.) John Naylor, reviewed and whacked around a bit by Dagfinn
Ilmari Mannsåker, and some more by me. Discussion:

- Add missing pg_description strings for pg_type entries. I noticed that all
non-composite, non-array entries in pg_type.dat had descr strings, except for
"json" and the pseudo-types. The lack for json seems certainly an oversight,
and there's surely little reason to not have entries for the pseudo-types
either. So add some. "make reformat-dat-files" turned up some formatting
issues in pg_amop.dat, too, so fix those in passing. No catversion bump since
the backend doesn't care too much what is in pg_description.

- Add a "return" statement to pacify perlcritic. Per buildfarm member crake.

- Fix psql's tab completion for TABLE. This should offer the same relation
types that SELECT ... FROM would. You can't select from an index for
instance, so offering it here is unhelpful. Noted while testing ilmari's
recent patch.

- Fix psql's tab completion for ALTER DATABASE ... SET TABLESPACE. We have the
infrastructure to offer a list of tablespace names, but it wasn't being used
here; instead you got "FROM", "CURRENT", and "TO" which aren't actually legal
in this syntax. Dagfinn Ilmari Mannsåker, reviewed by Arthur Zakirov
Discussion: https://postgr.es/m/d8jo9djvm7h.fsf@dalvik.ping.uio.no

- Rationalize Query_for_list_of_[relations] query names in tab-complete.c. The
previous convention was to use names based on the set of relkinds being
selected for, which was not at all helpful for maintenance, especially since
people had been quite inconsistent about whether to change the names when they
changed the relkinds being selected for. Instead, use names based on the
functionality we need the relation to have, following the model established by
Query_for_list_of_updatables. While at it, sort the list of Query constants a
bit better; it had the distinct air of code-assembled-by-dartboard before.
Discussion: https://postgr.es/m/14830.1537481254@sss.pgh.pa.us

- Improve tab completion for ANALYZE, EXPLAIN, and VACUUM. Previously, we made
no attempt to provide tab completion in these statements' optional
parenthesized options lists. This patch teaches psql to do so. To prevent
the option completions from being offered after we've already seen a complete
parenthesized option list, it's necessary to improve word_matches() so that it
allows a wildcard '*' in the middle of an alternative, not only at the end as
formerly. That requires only a little more code than before, and it allows us
to test for "incomplete parenthesized options" with a test like else if
(HeadMatches2("EXPLAIN", "(*") && !HeadMatches2("EXPLAIN", "(*)")) In
addition, add some logic to offer column names in the context of "ANALYZE
tablename ( ...", and likewise for VACUUM. This isn't real complete; it won't
offer column names again after a comma. But it's better than before, and it
doesn't take much code. Justin Pryzby, reviewed at various times by Álvaro
Herrera, Arthur Zakirov, and Edmund Horner; some additional fixups by me
Discussion: https://postgr.es/m/20180529000623.GA21896@telsasoft.com

- Fix bogus tab-completion rule for CREATE PUBLICATION. You can't use "FOR
TABLE" as a single Matches argument, because readline will consider that input
to be two words not one. It's necessary to make the pattern contain two
arguments. The case accidentally worked anyway because the words_after_create
test fired ... but only for the first such table name. Noted by Edmund
Horner, though this isn't exactly his proposed fix. Backpatch to v10 where
the faulty code came in. Discussion:

- Get rid of explicit argument-count markings in tab-complete.c. This replaces
the "TailMatchesN" macros with just "TailMatches", and likewise "HeadMatchesN"
becomes "HeadMatches" and "MatchesN" becomes "Matches". The various
COMPLETE_WITH_LISTn macros are reduced to COMPLETE_WITH, and the single-item
COMPLETE_WITH_CONST also gets folded into that. This eliminates a lot of
minor annoyance in writing tab-completion rules. Usefully, the compiled code
also gets a bit smaller (10% or so, on my machine). The implementation
depends on variadic macros, so we couldn't have done this before we required
C99. Andres Freund and Thomas Munro; some cosmetic cleanup by me.
Discussion: https://postgr.es/m/d8jo9djvm7h.fsf@dalvik.ping.uio.no

Alexander Korotkov pushed:

- Add support for nearest-neighbor (KNN) searches to SP-GiST. Currently, KNN
searches were supported only by GiST. SP-GiST also capable to support them.
This commit implements that support. SP-GiST scan stack is replaced with
queue, which serves as stack if no ordering is specified. KNN support is
provided for three SP-GIST opclasses: quad_point_ops, kd_point_ops and
poly_ops (catversion is bumped). Some common parts between GiST and SP-GiST
KNNs are extracted into separate functions. Discussion:
Author: Nikita Glukhov, Alexander Korotkov based on GSoC work by Vlad
Sterzhanov Review: Andrey Borodin, Alexander Korotkov

- Fix handling of format string text characters in to_timestamp()/to_date().
cf984672 introduced improvement of handling of spaces and separators in
to_timestamp()/to_date() functions. In particular, now we're skipping spaces
both before and after fields. That may cause format string text character to
consume part of field in the situations, when it didn't happen before
cf984672. This commit cause format string text character consume input string
characters only when since previous field (or string beginning) number of
skipped input string characters is not greater than number of corresponding
format string characters (that is we didn't skip any extra characters in input

- Replace CAS loop with single TAS in ProcArrayGroupClearXid(). Single
pg_atomic_exchange_u32() is expected to be faster than loop of
pg_atomic_compare_exchange_u32(). Also, it would be consistent with clog
group update code. Discussion:
Reviewed-by: Amit Kapila

Andres Freund pushed:

- Error out for clang on x86-32 without SSE2 support, no -fexcess-precision. As
clang currently doesn't support -fexcess-precision=standard, compiling x86-32
code with SSE2 disabled, can lead to problems with floating point overflow
checks and the like. This issue was noticed because clang, on at least some
BSDs, defaults to i386 compatibility, whereas it defaults to pentium4 on
Linux. Our forced usage of __builtin_isinf() lead to some overflow checks not
triggering when compiling for i386, e.g. when the result of the calculation
didn't overflow in 80bit registers, but did so in 64bit. While we could just
fall back to a non-builtin isinf, it seems likely that the use of 80bit
registers leads to other problems (which is why we force the flag for GCC
already). Therefore error out when detecting clang in that situation.
Reported-By: Victor Wagner Analyzed-By: Andrew Gierth and Andres Freund
Author: Andres Freund Discussion:
Backpatch: 9.3-, all supported versions are affected

Bruce Momjian pushed:

- docs: remove use of escape strings and use bytea hex output.
standard_conforming_strings defaulted to 'on' in PG 9.1. bytea_output
defaulted to 'hex' in PG 9.0. Reported-by: André Hänsel Discussion:
Backpatch-through: 9.3

- doc: JIT is enabled by default in PG 12. JIT was disabled by default in a PG
11 in a separate commit that will normally not appear in the PG 12 git logs.
Therefore, create a PG 12 document and mention the fact that JIT is enabled by
default in this release. (A similar change in parallelism was missed in a
prior release.) Reported-by: Andres Freund Discussion:
Backpatch-through: head

== Pending Patches ==

Amul Sul sent in a patch to ensure that CREATE TABLE ... PARTITION OF checks
that there isn't already a primary key.

Michaël Paquier sent in another revision of a patch to extend the lookup
routines for FDW and foreign server with NULL handling and eliminate
user-visible cache lookup errors for objaddr SQL functions.

Michael Banck sent in a patch to swap out the pg_sleep() with a check against
the checkpoint LSN on a recheck verification failure.

Michael Banck sent in two more revisions of a patch to enable verifying page
checksums in a running cluster.

Kyotaro HORIGUCHI and Michaël Paquier traded patches to fix an issue that could
cause a crash when setting full page writes with a SIGHUP.

Thomas Munro sent in another revision of a patch to use pread()/pwrite() instead
of lseek() + read()/write().

Thomas Munro sent in another revision of a patch to track the next xid using 64

Peter Geoghegan sent in another revision of a patch to make all nbtree entries
unique by having heap TIDs participate in comparisons.

John Naylor sent in a patch to add a description for acldefault in pg_proc.dat.

Joe Conway sent in two more revisions of a patch to add missing documentation
for "acldefault".

Pavel Stěhule sent in three more revisions of a patch to implement schema

Alexander Korotkov sent in a patch to document the new handling of letters and
digits in to_date() and to_timestamp().

Adrien Nayrat sent in a patch to include "Push LIMIT through subqueries to
underlying sort" in the release notes for 11.

Masahiko Sawada sent in another revision of a patch to add a commitTS recovery

Andrey V. Lepikhov sent in a patch to implement a quick vacuum strategy.

Andrey V. Lepikhov sent in another revision of a patch to add a retail
IndexTuple Deletion Access Method.

Laurenz Albe sent in a patch to add pg_promote() to promote standby servers.

Chris Travers sent in a patch to refactor signal detection.

Pavel Stěhule sent in another revision of a patch to add default namespaces for
XPath expressions.

Alexander Kukushkin sent in another revision of a patch to implement a new GUC,

Thomas Munro sent in two more revisions of a patch to enable SERIALIZABLE READ
ONLY DEFERRABLE isolation for hot standbys.

Gilles Darold sent in a patch to fix a bug in pg_dump/pg_restore with the
--no-publication option.

Don Seiler sent in a patch to application_name to the Port struct to make it
possible to display same in "connection authorized" log messages.

Edmund Horner sent in another revision of a patch to add tab completion for
SELECT to psql.

Tom Lane sent in two more revisions of a patch to (optionally) build pgbench
using ppoll() for larger connection counts.

Mark G sent in a patch to make dsa_size_class_map const.

Andrey Borodin sent in a patch to add a GiST verification function for amcheck.

Keiichi Hirobe sent in a patch to fix a bug where too much memory was allocated
for curly braces in array_out().

Thomas Munro sent in a patch to track versions of collations.

Browse pgsql-announce by date

  From Date Subject
Next Message Regina Obe 2018-09-24 16:26:58 PostGIS 2.5.0 released
Previous Message Nicolas Thauvin 2018-09-21 14:03:20 pitrery 2.2 released