== PostgreSQL Weekly News - July 29 2018 ==

From: David Fetter <david(at)fetter(dot)org>
To: PostgreSQL Announce <pgsql-announce(at)postgresql(dot)org>
Subject: == PostgreSQL Weekly News - July 29 2018 ==
Date: 2018-07-29 22:52:32
Message-ID: 20180729225232.GA4628@fetter.org
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-announce

== PostgreSQL Weekly News - July 29 2018 ==

== PostgreSQL Jobs for July ==


== PostgreSQL Local ==

PGConf.Brazil 2018 will take place in São Paulo, Brazil on August 3-4 2018.

PostgreOpen Silicon Valley 2018 will be held in San Francisco on September 5-7, 2018.

The Portland PostgreSQL Users Group will be holding a PGDay on September 10,
2018 in Portland, OR. The CfP is open at https://goo.gl/forms/E0CiUQGSZGMYwh922

PostgresConf South Africa 2018 will take place in Johannesburg on October 9, 2018

PostgreSQL Conference Europe 2018 will be held on October 23-26, 2018 at the
Lisbon Marriott Hotel in Lisbon, Portugal. The CfP is open through August 6,
2018 midnight CET at https://2018.pgconf.eu/callforpapers

2Q PGConf will be on December 4-5, 2018 in Chicago, IL. The CfP is open through
August 27, 2018 at midnight Pacific Time at http://www.2qpgconf.com/#cfp

PGConf.ASIA 2018 will take place on December 10-12, 2018 in Akihabara, Tokyo,
Japan. The CfP is open until midnight, July 31, 2018, Japan time at

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Andres Freund pushed:

- Fix JITed EEOP_AGG_INIT_TRANS, which missed some state. The JIT compiled
implementation missed maintaining AggState->{current_set,curaggcontext}. That
could lead to trouble because the transition value could be allocated in the
wrong context. Reported-By: Rushabh Lathia Diagnosed-By: Dmitry Dolgov
Author: Dmitry Dolgov, with minor changes by me Discussion:
Backpatch: 11-, where JIT compilation support was added

- Mop-up for 3522d0eaba5, which missed some alternative output files.

- Reset context at the tail end of JITed EEOP_AGG_PLAIN_TRANS. While no
negative consequences are currently known, it's clearly wrong to not reset the
context in one of the branches. Reported-By: Dmitry Dolgov Author: Dmitry
Dolgov Discussion:
Backpatch: 11-, where JIT compilation support was added

- LLVMJIT: Fix LLVM build for LLVM > 7. The location of
LLVMAddPromoteMemoryToRegisterPass moved. Author: Andres Freund Backpatch:
11, where LLVM based JIT support was added.

- LLVMJIT: Adapt to API changes in gdb and perf support. During the work of
upstreaming my previous patches for gdb and perf support the API changed.
Adapt. Normally this wouldn't necessarily be something to backpatch, but the
previous API wasn't upstream, and at least the gdb support is quite useful for
debugging. Author: Andres Freund Backpatch: 11, where LLVM based JIT support
was added.

- Defend against some potential spurious compiler warnings in 86eaf208e.
Author: David Rowley Discussion:

- doc: Fix reference to "decoder" to instead be the correct "output plugin".
Author: Jonathan Katz Discussion:
Backpatch: 9.4-, where logical decoding was added

- LLVMJIT: Check for 'noinline' attribute in recursively inlined functions.
Previously the attribute was only checked for external functions inlined, not
"static" functions that had to be inlined as dependencies. This isn't really
a bug, but makes debugging a bit harder. The new behaviour also makes more
sense. Therefore backpatch. Author: Andres Freund Backpatch: 11-, where JIT
compilation was added

- LLVMJIT: Release JIT context after running ExprContext shutdown callbacks.
Due to inlining it previously was possible that an ExprContext's shutdown
callback pointed to a JITed function. As the JIT context previously was shut
down before the shutdown callbacks were called, that could lead to segfaults.
Fix the ordering. Reported-By: Dmitry Dolgov Author: Andres Freund
Backpatch: 11-, where JIT compilation was added

Michaël Paquier pushed:

- Make more consistent some error messages for file-related operations. Some
error messages which report something about a file operation use as well
context which is already provided within the path being worked on, making
things rather duplicated. This creates more work for translators, and does
not actually bring clarity. More could be done, however in a lot of cases the
context used is actually useful, still that patch gets down things with a good
cut. Author: Michael Paquier Reviewed-by: Kyotaro Horiguchi, Tom Lane
Discussion: https://postgr.es/m/20180718044711.GA8565@paquier.xyz

- Add proper errcodes to new error messages for read() failures. Those would
use the default ERRCODE_INTERNAL_ERROR, but for foreseeable failures an
errcode ought to be set, ERRCODE_DATA_CORRUPTED making the most sense here.
While on the way, fix one errcode_for_file_access missing in origin.c since
the code has been created, and remove one assignment of errno to 0 before
calling read(), as this was around to fit with what was present before
811b6e36 where errno would not be set when not enough bytes are read. I have
noticed the first one, and Tom has pinged me about the second one. Author:
Michael Paquier Reported-by: Tom Lane Discussion:

- Fix calculation for WAL segment recycling and removal. Commit 4b0d28de06 has
removed the prior checkpoint and related facilities but has left WAL recycling
based on the LSN of the prior checkpoint, which causes incorrect calculations
for WAL removal and recycling for max_wal_size and min_wal_size. This commit
changes things so as the base calculation point is the last checkpoint
generated. Reported-by: Kyotaro Horiguchi Author: Kyotaro Horiguchi
Reviewed-by: Michael Paquier Discussion:
Backpatch: 11-, where the prior checkpoint has been removed.

- Refactor cluster_rel() to handle more options. This extends cluster_rel() in
such a way that more options can be added in the future, which will reduce the
amount of chunk code for an upcoming SKIP_LOCKED aimed for VACUUM. As VACUUM
FULL is a different flavor of CLUSTER, we want to make that extensible to ease
integration. This only reworks the API and its callers, without providing
anything user-facing. Two options are present now: verbose mode and relation
recheck when doing the cluster command work across multiple transactions.
This could be used as well as a base to extend the grammar of CLUSTER later
on. Author: Michael Paquier Reviewed-by: Nathan Bossart Discussion:

- Fix handling of pgbench's hash when no argument is provided. Depending on the
platform used, this can cause a crash in the worst case, or an unhelpful error
message, so fail gracefully. Author: Fabien Coelho Discussion:
https://postgr.es/m/alpine.DEB.2.21.1807262302550.29874@lancre Backpatch: 11-,
where hash() has been added in pgbench.

- Add verbosity to pg_basebackup for sync. This is useful to know when the data
copy has been finished. The current situation can be confusing for users as
the last message is "waiting for background process to finish streaming", so
it looks like this is taking time but the final sync is instead. Author: Jeff
Janes Discussion:

- Fix two oversights from 9ebe0572 which refactored cluster_rel. The recheck
option became a no-op as ClusterOption failed to set proper values for each
element. There was a second code path where local options got overwritten.
Both issues have been spotted by Coverity.

- Make error message of pageinspect more consistent for raw page inputs. There
is a copy-paste error from bt_page_items() which got into
bt_page_items_bytea(). A second message in get_raw_page_internal() was
inconsistent with all the other sub-modules. Author: Ashutosh Sharma

Robert Haas pushed:

- pgbench: Remove duplicate entries from table of builtin functions. Fabien
Coelho Discussion:

- Use key and partdesc from PartitionDispatch where possible. Instead of
repeatedly fishing the data out of the relcache entry, let's use the version
that we cached in the PartitionDispatch. We could alternatively rip out the
PartitionDispatch fields altogether, but it doesn't make much sense to have
them and not use them; before this patch, partdesc was set but altogether
unused. Amit Langote and I both thought using them was a litle better than
removing them, so this patch takes that approach. Discussion:

Thomas Munro pushed:

- Use setproctitle_fast() to update the ps status, if available. FreeBSD has
introduced a faster variant of setproctitle(). Use it, where available.
Author: Thomas Munro Discussion:

- Pad semaphores to avoid false sharing. In a USE_UNNAMED_SEMAPHORES build, the
default on Linux and FreeBSD since commit ecb0d20a, we have an array of sem_t
objects. This turned out to reduce performance compared to the previous
default USE_SYSV_SEMAPHORES on an 8 socket system. Testing showed that the
lost performance could be regained by padding the array elements so that they
have their own cache lines. This matches what we do for similar hot arrays
(see LWLockPadded, WALInsertLockPadded). Back-patch to 10, where unnamed
semaphores were adopted as the default semaphore interface on those operating
systems. Author: Thomas Munro Reviewed-by: Andres Freund Reported-by: Mithun
Cy Tested-by: Mithun Cy, Tom Lane, Thomas Munro Discussion:

Peter Eisentraut pushed:

- psql: Add option for procedures to \df.

Tomáš Vondra:

- Add strict_multi_assignment and too_many_rows plpgsql checks. Until now
shadowed_variables was the only plpgsql check supported by
plpgsql.extra_warnings and plpgsql.extra_errors. This patch introduces two
new checks - strict_multi_assignment and too_many_rows. Unlike
shadowed_variables, these new checks are enforced at run-time.
strict_multi_assignment checks that commands allowing multi-assignment (for
example SELECT INTO) have the same number of sources and targets.
too_many_rows checks that queries with an INTO clause return one row exactly.
These checks are aimed at cases that are technically valid and allowed, but
are often a sign of a bug. Therefore those checks are expected to be enabled
primarily in development and testing environments. Author: Pavel Stehule
Reviewed-by: Stephen Frost, Tomas Vondra Discussion:

- Provide separate header file for built-in float types. Some data types under
adt/ have separate header files, but most simple ones do not, and their public
functions are defined in builtins.h. As the patches improving geometric types
will require making additional functions public, this seems like a good
opportunity to create a header for floats types. Commit 1acf757255 made _cmp
functions public to solve NaN issues locally for GiST indexes. This patch
reworks it in favour of a more widely applicable API. The API uses inline
functions, as they are easier to use compared to macros, and avoid
double-evaluation hazards. Author: Emre Hasegeli Reviewed-by: Kyotaro
Horiguchi Discussion:

- Refactor geometric functions and operators. The primary goal of this patch is
to eliminate duplicate code and share code between different geometric data
types more often, to prepare the ground for additional patches. Until now the
code reuse was limited, probably because the simpler types (line and point)
were implemented after the more complex ones. The changes are quite extensive
and can be summarised as: * Eliminate SQL-level function calls. * Re-use more
functions to implement others. * Unify internal function names and
signatures. * Remove private functions from geo_decls.h. * Replace
should-not-happen checks with assertions. * Add comments describe for various
functions. * Remove some unreachable code. * Define delimiter symbols of
line datatype like the other ones. * Remove the GEODEBUG macro and printf()
calls. * Unify code style of a few oddly formatted lines. While the goal was
to cause minimal user-visible changes, it was not possible to keep the
original behavior in all cases - for example when handling NaN values, or when
reusing code makes the functions return consistent results. Author: Emre
Hasegeli Reviewed-by: Kyotaro Horiguchi, me Discussion:

- Restore handling of -0 in the C field of lines in line_construct(). Commit
a7dc63d904 inadvertedly removed this bit originally introduced by 43fe90f66a,
causing regression test failures on some platforms, due to producing {1,-1,-0}
instead of {1,-1,0}.

Tom Lane pushed:

- Avoid crash in eval_const_expressions if a Param's type changes. Since commit
6719b238e it's been possible for the values of plpgsql record field variables
to be exposed to the planner as Params. (Before that, plpgsql never supplied
values for such variables during planning, so that the problematic code wasn't
reached.) Other places that touch potentially-type-mutable Params either cope
gracefully or do runtime-test-and-ereport checks that the type is what they
expect. But eval_const_expressions() just had an Assert, meaning that it
either failed the assertion or risked crashes due to using an incompatible
value. In this case, rather than throwing an ereport immediately, we can just
not perform a const-substitution in case of a mismatch. This seems important
for the same reason that the Param fetch was speculative: we might not
actually reach this part of the expression at runtime. Test case will follow
in a separate commit. Patch by me, pursuant to bug report from Andrew Gierth.
Back-patch to v11 where the previous commit appeared. Discussion:

- Provide plpgsql tests for cases involving record field changes. We suppressed
one of these test cases in commit feb1cc559 because it was failing to produce
the expected results on CLOBBER_CACHE_ALWAYS buildfarm members. But now we
need another test with similar behavior, so let's set up a test file that is
expected to vary between regular and CLOBBER_CACHE_ALWAYS cases, and provide
variant expected files. Someday we should fix plpgsql's failure for
change-of-field-type, and then the discrepancy will go away and we can fold
these tests back into plpgsql_record.sql. But today is not that day.
Discussion: https://postgr.es/m/87wotkfju1.fsf@news-spur.riddles.org.uk

Amit Kapila pushed:

- Fix the buffer release order for parallel index scans. During parallel index
scans, if the current page to be read is deleted, we skip it and try to get
the next page for a scan without releasing the buffer lock on the current
page. To get the next page, sometimes it needs to wait for another process to
complete its scan and advance it to the next page. Now, it is quite possible
that the master backend has errored out before advancing the scan and issued a
termination signal for all workers. The workers failed to notice the
termination request during wait because the interrupts are held due to buffer
lock on the previous page. This lead to all workers being stuck. The fix is
to release the buffer lock on current page before trying to get the next page.
We are already doing same in backward scans, but missed it for forward scans.
Reported-by: Victor Yegorov Bug: 15290 Diagnosed-by: Thomas Munro and Amit
Kapila Author: Amit Kapila Reviewed-by: Thomas Munro Tested-By: Thomas Munro
and Victor Yegorov Backpatch-through: 10 where parallel index scans were
introduced Discussion:

Álvaro Herrera pushed:

- Fix grammar in README.tuplock. Author: Brad DeJong Discussion:

Alexander Korotkov pushed:

- Reduce path length for locking leaf B-tree pages during insertion. In our
B-tree implementation appropriate leaf page for new tuple insertion is
acquired using _bt_search() function. This function always returns leaf page
locked in shared mode. In order to obtain exclusive lock, caller have to
relock the page. This commit makes _bt_search() function lock leaf page
immediately in exclusive mode when needed. That removes unnecessary relock
and, in turn reduces lock contention for B-tree leaf pages. Our experiments
on multi-core systems showed acceleration up to 4.5 times in corner case.
Author: Alexander Korotkov Reviewed-by: Yoshikazu Imai, Simon Riggs, Peter

Bruce Momjian pushed:

- pgtest: grab possible warnings from install.log. Since PG 9.5, 'make check'
records the build output in install.log, so look in there for warnings too.
Backpatch-through: 9.5

- pg_upgrade: report database names with missing extension libs. Previously
only the missing library name was reported, forcing users to look in all
databases to find the library entries. Discussion:
https://postgr.es/m/20180713162815.GA3835@momjian.us Author: Daniel
Gustafsson, me

- pg_upgrade: check for clean server shutdowns. Previously pg_upgrade checked
for the pid file and started/stopped the server to force a clean shutdown.
However, "pg_ctl -m immediate" removes the pid file but doesn't do a clean
shutdown, so check pg_controldata for a clean shutdown too. Diagnosed-by:
Vimalraj A Discussion:
Backpatch-through: 9.3

- pgtest: run clean, build, and check stages separately. This allows for
cleaner error reporting. Backpatch-through: 9.5

Noah Misch pushed:

- Document security implications of qualified names. Commit
5770172cb0c9df9e6ce27c507b449557e5b45124 documented secure schema usage, and
that advice suffices for using unqualified names securely. Document, in
typeconv-func primarily, the additional issues that arise with qualified
names. Back-patch to 9.3 (all supported versions). Reviewed by Jonathan S.
Katz. Discussion:

- Fix earthdistance test suite function name typo. Affected test queries have
been testing the wrong thing since their introduction in commit
4c1383efd132e4f532213c8a8cc63a455f55e344. Back-patch to 9.3 (all supported

== Pending Patches ==

Amit Langote sent in a patch to check the enable_partition_pruning setting in
the executor.

Kyotaro HORIGUCHI sent in another revision of a patch to add a capability to
limit the number of segments kept per replication slot by a GUC variable, add a
monitoring aid for max_slot_wal_keep_size, add a TAP test for the slot limit
feature, and document the max_slot_wal_keep_size feature.

Kyotaro HORIGUCHI sent in two more revisions of a patch to fix the base used to
calculate WAL recycling.

Andrey V. Lepikhov sent in another revision of a patch to implement retail
IndexTuple deletion.

Michaël Paquier sent in a patch to refactor ClusterStmt to handle more options.

Pavel Stěhule sent in a patch to add a \dP option to psql to handle partitioned

Sergei Kornilov sent in another revision of a patch to log prepared statement

Fabien COELHO sent in a patch to pgbench to improve --help and --version

임명규 (Myungkyu Lim) sent in a patch to add accumulated statistics for wait

Andrey Borodin sent in two more revisions of a patch to implement GiST VACUUM.

Thomas Munro sent in a patch to creates the basic infrastructure to adopt 64 bit
transaction IDs as appropriate, and uses same to fix a bug where an epoch
increment could be missed if over 2^32 transactions occurred between

Chris Travers sent in a patch to document the fact that queries can be run over
the replication protocol.

Aleksandr Parfenov sent in another revision of a patch to make a more flexible
configuration for full-text search.

KaiGai Kohei and Amit Langote traded patches to fix a bug which caused memory
leaks in COPY FROM on partitioned tables.

Haribabu Kommi sent in a patch to merge tableam.h and tableamapi.h and create a
new API to get heap page tuples.

David Rowley sent in a patch to ensure that pruning steps are only built when
all 'subpaths' have leaf partition parents.

David Cramer sent in another revision of a patch to respect client-initiated
CopyDone during logical streaming replication.

Nathan Bossart sent in a patch to refactor the logging logic for skipped
relations in VACUUM and ANALYZE.

Daniel Gustafsson sent in another revision of a patch to refactor the backend
signalling code and use same to support an optional message in backend

Amit Langote sent in a patch to fix a bug with how
ExecFindInitialMatchingSubPlans handles other_subplans where while the
subplan_map arrays are updated to contain revised values after pruning, those in
the other_subplans Bitmapset were not.

Arseny Sher sent in another revision of a patch to implement global snapshot
using CSN.

Antonin Houska sent in another revision of a patch to implement data-at-rest

Cynthia Shang sent in a patch to move some copy tests around to accommodate the
new COPY option for the text format which enables adding a header line.

Daniel Gustafsson sent in another revision of a patch to enable checksums

Robert Haas sent in a series of patches intended to center information about
tuple routing in the PartitionDispatch structure.

Andrew Gierth and David Fetter traded patches to inline non-recursive WITH
clauses for performance.

Daniel Gustafsson sent in another revision of a patch to order windows on
partition/ordering prefix to reuse Sort nodes.

Amit Langote sent in a patch to disallow domain type partition key.

Ashutosh Bapat sent in another revision of a patch to create a TableTupleSlot

Simon Muller sent in another revision of a patch to add an option to COPY's
'text' format to output a header.

Tom Lane sent in a patch to fix a 11beta crash/assert caused by parameter type

Amit Langote sent in two more revisions of a patch to add partition tree
inspection functions.

Nikhil Sontakke sent in another revision of a patch to implement logical
decoding of two-phase transactions.

David Rowley and Amit Langote traded patches to speed up INSERTs and UPDATEs to
partitioned tables.

David Rowley sent in two more revisions of a patch to allow multi-inserts during
COPY into a partitioned table.

Michaël Paquier sent in two more revisions of a patch to refactor TRUNCATE and VACUUM
execution to avoid early lock lookups, and restrict access to system-wide
REINDEX for non-privileged users.

Thomas Munro sent in another revision of a patch to use pread()/pwrite() instead
of lseek() + read()/write().

Amit Langote sent in two revisions of a patch to fix a bounds check for
bitmapset members for partition pruning.

Michaël Paquier sent in a patch to prevent temp tables from preventing
autovacuum, which would eventually cause xid wraparound.

Fabien COELHO sent in a patch to ensure that pgbench's hash() fails gracefully.

Alexander Kuzmenkov sent in another revision of a patch to order windows on
partition/ordering prefix to reuse Sort nodes.

Alexander Kuzmenkov sent in another revision of a patch to remove unneeded

Jeff Janes sent in a patch to add verbosity to pg_basebackup for sync.

Nathan Bossart sent in another revision of a patch to fix BUG #15182.

Amit Kapila sent in a patch to fix gather stats.

Fabien COELHO sent in a patch to pgbench to add a pseudo-random permutation

Álvaro Herrera sent in a patch to fix a crash in logical replication subscriber.

David Fetter sent in another revision of a patch to add RESPECT/IGNORE NULLS and
FROM FIRST/LAST options to windowing functions.

Amit Kapila sent in a patch to attach FPI to the first record after
full_page_writes is turned on.

David Fetter sent in a patch to fix an issue in psql where \dp/\z displayed
default permissions the same way it did total lack of permissions.

Fabien COELHO sent in a patch to remove a legacy final "\." in pgbench.

Tom Lane sent in a patch to fix some bizarre behavior in libpq's searching of

Browse pgsql-announce by date

  From Date Subject
Next Message Tatsuo Ishii 2018-07-30 00:59:49 Re: Call for Papers - PGConf.ASIA 2018
Previous Message Tatsuo Ishii 2018-07-23 02:45:05 Re: Call for Papers - PGConf.ASIA 2018