== PostgreSQL Weekly News - September 30, 2018 ==

== PostgreSQL Weekly News - September 30, 2018 ==

PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy. The CfP
is open at https://2019.pgday.it/en/blog/cfp and the Call for Workshops is at
https://2019.pgday.it/en/blog/cfw until January 15, 2019.
https://2019.pgday.it/en/

pgDay Paris 2019 will be held in Paris, France on March 12, 2019
at 199bis rue Saint-Martin. The CfP is open until November 30, 2018.
http://2019.pgday.paris/callforpapers/

== PostgreSQL Product News ==

Ora2Pg 19.1, a tool for migrating Oracle databases to PostgreSQL, released.
http://ora2pg.darold.net/

PostGIS 2.5.0 the industry standard geographic information
system package for PostgreSQL, released.
http://postgis.net/2018/09/23/postgis-2.5.0/

== PostgreSQL Jobs for September ==

http://archives.postgresql.org/pgsql-jobs/2018-9/

== PostgreSQL Local ==

PostgresConf South Africa 2018 will take place in Johannesburg on October 9, 2018
https://postgresconf.org/conferences/SouthAfrica2018

PostgreSQL Conference Europe 2018 will be held on October 23-26, 2018 at the
Lisbon Marriott Hotel in Lisbon, Portugal.
https://2018.pgconf.eu/

2Q PGConf will be on December 4-5, 2018 in Chicago, IL.
http://www.2qpgconf.com/

PGConf.ASIA 2018 will take place on December 10-12, 2018 in Akihabara, Tokyo,
Japan.
http://www.pgconf.asia/EN/2018/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Fix failure in WHERE CURRENT OF after rewinding the referenced cursor. In a
case where we have multiple relation-scan nodes in a cursor plan, such as a
scan of an inheritance tree, it's possible to fetch from a given scan node,
then rewind the cursor and fetch some row from an earlier scan node. In such
a case, execCurrent.c mistakenly thought that the later scan node was still
active, because ExecReScan hadn't done anything to make it look not-active.
We'd get some sort of failure in the case of a SeqScan node, because the
node's scan tuple slot would be pointing at a HeapTuple whose t_self gets
reset to invalid by heapam.c. But it seems possible that for other relation
scan node types we'd actually return a valid tuple TID to the caller,
resulting in updating or deleting a tuple that shouldn't have been considered
current. To fix, forcibly clear the ScanTupleSlot in ExecScanReScan. Another
issue here, which seems only latent at the moment but could easily become a
live bug in future, is that rewinding a cursor does not necessarily lead to
*immediately* applying ExecReScan to every scan-level node in the plan tree.
Upper-level nodes will think that they can postpone that call if their child
node is already marked with chgParam flags. I don't see a way for that to
happen today in a plan tree that's simple enough for execCurrent.c's
search_plan_tree to understand, but that's one heck of a fragile assumption.
So, add some logic in search_plan_tree to detect chgParam flags being set on
nodes that it descended to/through, and assume that that means we should
consider lower scan nodes to be logically reset even if their ReScan call
hasn't actually happened yet. Per bug #15395 from Matvey Arye. This has been
broken for a long time, so back-patch to all supported branches. Discussion:
https://postgr.es/m/153764171023.14986.280404050547008575@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/89b280e139c463c98eb33592216a123e89052d08

- Doc: warn against using parallel restore with --load-via-partition-root. This
isn't terribly safe, and making it so doesn't seem like a small project, so
for the moment just warn against it. Discussion:
https://postgr.es/m/13624.1535486019@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/73a60051379b35a0bec399edfe369c59e50cc775

- Fix over-allocation of space for array_out()'s result string. array_out
overestimated the space needed for its output, possibly by a very substantial
amount if the array is multi-dimensional, because of wrong order of operations
in the loop that counts the number of curly-brace pairs needed. While the
output string is normally short-lived, this could still cause problems in
extreme cases. An additional minor error was that it counted one more
delimiter than is actually needed. Repair those errors, add an Assert that
the space is now correctly calculated, and make some minor improvements in the
comments. I also failed to resist the temptation to get rid of an integer
modulus operation per array element; a simple comparison is sufficient. This
bug dates clear back to Berkeley days, so back-patch to all supported
versions. Keiichi Hirobe, minor additional work by me Discussion:
https://postgr.es/m/CAH=EFxE9W0tRvQkixR2XJRRCToUYUEDkJZk6tnADXugPBRdcdg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/87d9bbca13f9c6b8f6ee986f0e399cb83bd731d4

- Use ppoll(2), if available, to wait for input in pgbench. Previously, pgbench
always used select(2) for this purpose, but that's problematic for very high
client counts, because select() can't deal with file descriptor numbers larger
than FD_SETSIZE. It's pretty common for that to be only 1024 or so, whereas
modern OSes can allow many more open files than that. Using poll(2) would
surmount that problem, but it creates another one: poll()'s timeout resolution
is only 1ms, which is poor enough to cause problems with --rate specifications
approaching or exceeding 1K TPS. On platforms that have ppoll(2), which
includes Linux and recent FreeBSD, we can use that to avoid the FD_SETSIZE
problem without any loss of timeout resolution. Hence, add configure logic to
test for ppoll(), and use it if available. This patch introduces an
abstraction layer into pgbench that could be extended to support other kernel
event-wait APIs such as kevents. But actually adding such support is a matter
for some future patch. Doug Rady, reviewed by Robert Haas and Fabien Coelho,
and whacked around a good bit more by me Discussion:
https://postgr.es/m/23D017C9-81B7-484D-8490-FD94DEC4DF59@amazon.com
https://git.postgresql.org/pg/commitdiff/60e612b602999e670f2d57a01e52799eaa903ca9

- Sync our Snowball stemmer dictionaries with current upstream. We haven't
touched these since text search functionality landed in core in 2007 :-(.
While the upstream project isn't a beehive of activity, they do make additions
and bug fixes from time to time. Update our copies of these files. Also
update our documentation about how to keep things in sync, since they're not
making distribution tarballs these days. Fortunately, their source code turns
out to be a breeze to build. Notable changes: * The non-UTF8 version of the
hungarian stemmer now works in LATIN2 not LATIN1. * New stemmers have
appeared for arabic, indonesian, irish, lithuanian, nepali, and tamil. These
all work in UTF8, and the indonesian and irish ones also work in LATIN1.
(There are some new stemmers that I did not incorporate, mainly because their
names don't match the underlying languages, suggesting that they're not to be
considered mainstream.) Worth noting: the upstream Nepali dictionary was
contributed by Arthur Zakirov. initdb forced because the contents of
snowball_create.sql have changed. Still TODO: see about updating the stopword
lists. Arthur Zakirov, minor mods and doc work by me Discussion:
https://postgr.es/m/20180626122025.GA12647@zakirov.localdomain Discussion:
https://postgr.es/m/20180219140849.GA9050@zakirov.localdomain
https://git.postgresql.org/pg/commitdiff/fd582317e10e26083b8c720598bfcdbf89787112

- Avoid unnecessary precision loss for pgbench's --rate target. It's fairly
silly to truncate the throttle_delay to integer when the only math we ever do
with it requires converting back to double. Furthermore, given that people
are starting to complain about restrictions like only supporting 1K client
connections, I don't think we're very far away from situations where the
precision loss matters. As the code stood, for example, there's no difference
between --rate 100001 and --rate 111111; both get converted to throttle_delay
= 9. Somebody trying to run 100 threads and have each one dispatch around 1K
TPS would find this lack of precision rather surprising, especially since the
required per-thread delays are around 1ms, well within the timing precision of
modern systems.
https://git.postgresql.org/pg/commitdiff/5b7e036707ccd93506731da82a56b07023d13e30

- Make some fixes to allow building Postgres on macOS 10.14 ("Mojave"). Apple's
latest rearrangements of the system-supplied headers have broken building of
PL/Perl and PL/Tcl. The only practical way to fix PL/Tcl is to start using
the "-isysroot" compiler flag to point to SDK-supplied headers, as Apple
expects. We must also start distinguishing where to find Perl's headers from
where to find its shared library; but that seems like good cleanup anyway.
Extensions that formerly did something like -I$(perl_archlibexp)/CORE should
now do -I$(perl_includedir)/CORE instead. perl_archlibexp is still the place
to look for libperl.so, though. If for some reason you don't like the default
-isysroot setting, you can override that by setting PG_SYSROOT in configure's
arguments. I don't currently think people would need to do so, unless maybe
for cross-version build purposes. In addition, teach configure where to find
tclConfig.sh. Our traditional method of searching $auto_path hasn't worked
for the last couple of macOS releases, and it now seems clear that Apple's not
going to change that. The workaround of manually specifying --with-tclconfig
was annoying already, but Mojave's made it a lot more so because the sysroot
path now has to be included as well. Let's just wire the knowledge into
configure instead. To avoid breaking builds against non-default Tcl
installations (e.g. MacPorts) wherein the $auto_path method probably still
works, arrange to try the additional case only after all else has failed.
Back-patch to all supported versions, since at least the buildfarm cares about
that. The changes are set up to not do anything on macOS releases that are
old enough to not have functional sysroot trees.
https://git.postgresql.org/pg/commitdiff/5e22171310f8d7c82219a6b978440e5144e88683

- Convert elog.c's useful_strerror() into a globally-used strerror wrapper.
elog.c has long had a private strerror wrapper that handles assorted possible
failures or deficiencies of the platform's strerror. On Windows, it also
knows how to translate Winsock error codes, which the native strerror does
not. Move all this code into src/port/strerror.c and define strerror() as a
macro that invokes it, so that both our frontend and backend code will have
all of this behavior. I believe this constitutes an actual bug fix on
Windows, since AFAICS our frontend code did not report Winsock error codes
properly before this. However, the main point is to lay the groundwork for
implementing %m in src/port/snprintf.c: the behavior we want %m to have is
this one, not the native strerror's. Note that this throws away the prior use
of src/port/strerror.c, which was to implement strerror() on platforms lacking
it. That's been dead code for nigh twenty years now, since strerror() was
already required by C89. We should likewise cause strerror_r to use this
behavior, but I'll tackle that separately. Patch by me, reviewed by Michael
Paquier Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/26e9d4d4ef16b5e2be96319f89ea6ba7f63a4d73

- Always use our own versions of *printf(). We've spent an awful lot of effort
over the years in coping with platform-specific vagaries of the *printf family
of functions. Let's just forget all that mess and standardize on always using
src/port/snprintf.c. This gets rid of a lot of configure logic, and it will
allow a saner approach to dealing with %m (though actually changing that is
left for a follow-on patch). Preliminary performance testing suggests that as
it stands, snprintf.c is faster than the native printf functions for some
tasks on some platforms, and slower for other cases. A pending patch will
improve that, though cases with floating-point conversions will doubtless
remain slower unless we want to put a *lot* of effort into that. Still, we've
not observed that *printf is really a performance bottleneck for most
workloads, so I doubt this matters much. Patch by me, reviewed by Michael
Paquier Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/96bf88d52711ad3a0a4cc2d1d9cb0e2acab85e63

- Implement %m in src/port/snprintf.c, and teach elog.c to rely on that. I
started out with the idea that we needed to detect use of %m format specs in
contexts other than elog/ereport calls, because we couldn't rely on that
working in *printf calls. But a better answer is to fix things so that it
does work. Now that we're using snprintf.c all the time, we can implement %m
in that and we've fixed the problem. This requires also adjusting our various
printf-wrapping functions so that they ensure "errno" is preserved when they
call snprintf.c. Remove elog.c's handmade implementation of %m, and let it
rely on snprintf to support the feature. That should provide some performance
gain, though I've not attempted to measure it. There are a lot of places
where we could now simplify 'printf("%s", strerror(errno))' into
'printf("%m")', but I'm not in any big hurry to make that happen. Patch by
me, reviewed by Michael Paquier Discussion:
https://postgr.es/m/2975.1526862605@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/d6c55de1f99a9028540516316b95321a7b12a540

- Fix link failures due to snprintf/strerror changes. snprintf.c requires
isnan(), which requires -lm on some platforms. libpq never bothered with -lm
before, but now it needs it. strerror.c tries to translate a string or two,
which requires -lintl. We'd managed never to need that anywhere in
ecpg/pgtypeslib/ before, but now we do. Per buildfarm and a report from Peter
Eisentraut. Discussion:
https://postgr.es/m/20180926190934.ea4xvzhkayuw7gkx@alap3.anarazel.de
Discussion:
https://postgr.es/m/f67b5008-9f01-057f-2bff-558cb53af851@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/a6b88d682cbec73474a73c9782fb7096e9440a8b

- Clean up *printf macros to avoid conflict with format archetypes. We must
define the macro "printf" with arguments, else it can mess up format archetype
attributes in builds where PG_PRINTF_ATTRIBUTE is just "printf". Fortunately,
that's easy to do now that we're requiring C99; we can use __VA_ARGS__. On
the other hand, it's better not to use __VA_ARGS__ for the rest of the *printf
crew, so that one can take the addresses of those functions without surprises.
I'd proposed doing this some time ago, but forgot to make it happen; buildfarm
failures subsequent to 96bf88d52 reminded me. Discussion:
https://postgr.es/m/22709.1535135640@sss.pgh.pa.us Discussion:
https://postgr.es/m/20180926190934.ea4xvzhkayuw7gkx@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/8b91d258844afa58e856ac354f9ba9745ff9ffb2

- Try another way to detect the result type of strerror_r(). The method we've
traditionally used, of redeclaring strerror_r() to see if the compiler
complains of inconsistent declarations, turns out not to work reliably because
some compilers only report a warning, not an error. Amazingly, this has gone
undetected for years, even though it certainly breaks our detection of whether
strerror_r succeeded. Let's instead test whether the compiler will take the
result of strerror_r() as a switch() argument. It's possible this won't work
universally either, but it's the best idea I could come up with on the spur of
the moment. We should probably back-patch this once the dust settles, but
first let's see what the buildfarm thinks of it. Discussion:
https://postgr.es/m/10877.1537993279@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/751f532b9766fb5d3c334758abea95b7bb085c5a

- Fix another portability issue from commit 758ce9b77. strerror.c now requires
strlcpy() in some cases, and a couple of the ecpg libraries did not have that
at hand. Pull it in from src/port/ following the usual recipe. Per
buildfarm.
https://git.postgresql.org/pg/commitdiff/ce4887bd025b95c7b455fefd817a418844c6aad3

- Incorporate strerror_r() into src/port/snprintf.c, too. This provides the
features that used to exist in useful_strerror() for users of strerror_r(),
too. Also, standardize on the GNU convention that strerror_r returns a char
pointer that may not be NULL. I notice that libpq's win32.c contains a
variant version of strerror_r that probably ought to be folded into
strerror.c. But lacking a Windows environment, I should leave that to
somebody else. Discussion: https://postgr.es/m/2975.1526862605@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/758ce9b7794845f95473c569155d29fcf0e2751b

- Fix assorted bugs in pg_get_partition_constraintdef(). It failed if passed a
nonexistent relation OID, or one that was a non-heap relation, because of
blindly applying heap_open to a user-supplied OID. This is not OK behavior
for a SQL-exposed function; we have a project policy that we should return
NULL in such cases. Moreover, since pg_get_partition_constraintdef ought now
to work on indexes, restricting it to heaps is flat wrong anyway. The
underlying function generate_partition_qual() wasn't on board with indexes
having partition quals either, nor for that matter with rels having
relispartition set but yet null relpartbound. (One wonders whether the person
who wrote the function comment blocks claiming that these functions allow a
missing relpartbound had ever tested it.) Fix by testing relispartition before
opening the rel, and by using relation_open not heap_open. (If any other
relkinds ever grow the ability to have relispartition set, the code will work
with them automatically.) Also, don't reject null relpartbound in
generate_partition_qual. Back-patch to v11, and all but the null-relpartbound
change to v10. (It's not really necessary to change generate_partition_qual
at all in v10, but I thought s/heap_open/relation_open/ would be a good idea
anyway just to keep the code in sync with later branches.) Per report from
Justin Pryzby. Discussion:
https://postgr.es/m/20180927200020.GJ776@telsasoft.com
https://git.postgresql.org/pg/commitdiff/aaf10f32a308bc5f53772c773edf3f345f59bb74

- Build src/port files as a library with -fPIC, and use that in libpq. libpq
and ecpg need shared-library-friendly versions of assorted src/port/ and
src/common/ modules. Up to now, they got those by symlinking the individual
source files and compiling them locally. That's baroque, and a pain to
maintain, and it results in some amount of duplicated compile work. It
might've made sense when only a couple of files were needed, but the list has
grown and grown and grown :-( It makes more sense to have the originating
directory build a third variant of libpgport.a/libpgcommon.a containing
modules built with $(CFLAGS_SL), and just link that into the shared library.
Unused files won't get linked, so the end result should be the same. This
patch makes a down payment on that idea by having src/port/ build such a
library and making libpq use it. If the buildfarm doesn't expose fatal
problems with the approach, I'll extend it to the other cases. Discussion:
https://postgr.es/m/13022.1538003440@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/ea53100d5671b5b243f77898b0b04d23c38b2820

- Remove pqsignal() from libpq's official exports list. Client applications
should get this function, if they need it, from libpgport. The fact that it's
exported from libpq is a hack left over from before we set up libpgport. It's
never been documented, and there's no good reason for non-PG code to be
calling it anyway, so hopefully this won't cause any problems. Moreover, with
the previous setup it was not real clear whether our clients that use the
function were getting it from libpgport or libpq, so this might actually
prevent problems. The reason for changing it now is that in the wake of
commit ea53100d5, some linkers won't export the symbol, apparently because
it's coming from a .a library instead of a .o file. We could get around that
by continuing to symlink pqsignal.c into libpq as before; but unless somebody
complains very hard, I don't want to adopt such a kluge. Discussion:
https://postgr.es/m/13022.1538003440@sss.pgh.pa.us Discussion:
https://postgr.es/m/E1g5Y8r-0006vs-QA@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/f7ab802855200df5529a6e1e7b748d7926acace8

- Build src/common files as a library with -fPIC. Build a third version of
libpgcommon.a, with -fPIC and -DFRONTEND, as commit ea53100d5 did for
src/port. Use that in libpq to avoid symlinking+rebuilding source files
retail. Also adjust ecpg to use the new src/port and src/common libraries.
Arrange to install these libraries, too, to simplify out-of-tree builds of
shared libraries that need any of these modules. Discussion:
https://postgr.es/m/13022.1538003440@sss.pgh.pa.us Discussion:
https://postgr.es/m/E1g5Y8r-0006vs-QA@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/7143b3e82136d2941b3394168940d895a29b4f36

- Tweak MSVC build system to match changes in 7143b3e82. Also try to make the
comment suggesting that this might be needed more intelligible. Per
buildfarm.
https://git.postgresql.org/pg/commitdiff/97c6852ff7c54d5b44426ceae9e3215d13157c10

- Tweak MSVC build system to match changes in 7143b3e82. Looks like we need to
pull in $libpgcommon in a couple more places than before. Per buildfarm.
https://git.postgresql.org/pg/commitdiff/61f14cc8c85b5e6186c3b86c2c929821d7b33589

- Improve error reporting for unsupported effective_io_concurrency setting.
Give a specific error complaining about lack of posix_fadvise() when someone
tries to set effective_io_concurrency > 0 on platforms without that. This
probably isn't worth extensive back-patching, but I (tgl) felt cramming it
into v11 was reasonable. James Robinson Discussion:
https://postgr.es/m/153771876450.14994.560017943128223619@wrigleys.postgresql.org
Discussion:
https://postgr.es/m/A3942987-5BC7-4F05-B54D-2A0EC2914B33@jlr-photo.com
https://git.postgresql.org/pg/commitdiff/2b04dfc4724970231ac338aa71e529c823fc5ff6

- Create an RTE field to record the query's lock mode for each relation. Add
RangeTblEntry.rellockmode, which records the appropriate lock mode for each
RTE_RELATION rangetable entry (either AccessShareLock, RowShareLock, or
RowExclusiveLock depending on the RTE's role in the query). This patch
creates the field and makes all creators of RTE nodes fill it in reasonably,
but for the moment nothing much is done with it. The plan is to replace
assorted post-parser logic that re-determines the right lockmode to use with
simple uses of rte->rellockmode. For now, just add Asserts in each of those
places that the rellockmode matches what they are computing today. (In some
cases the match isn't perfect, so the Asserts are weaker than you might
expect; but this seems OK, as per discussion.) This passes check-world for me,
but it seems worth pushing in this state to see if the buildfarm finds any
problems in cases I failed to test. catversion bump due to change of stored
rules. Amit Langote, reviewed by David Rowley and Jesper Pedersen, and
whacked around a bit more by me Discussion:
https://postgr.es/m/468c85d9-540e-66a2-1dde-fec2b741e688@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/fdba460a26af919c0b366755d119f384706e670a

Noah Misch pushed:

- Initialize random() in bootstrap/stand-alone postgres and in initdb. This
removes a difference between the standard IsUnderPostmaster execution
environment and that of --boot and --single. In a stand-alone backend,
"SELECT random()" always started at the same seed. On a system capable of
using posix shared memory, initdb could still conclude "selecting dynamic
shared memory implementation ... sysv". Crashed --boot or --single postgres
processes orphaned shared memory objects having names that collided with the
not-actually-random names that initdb probed. The sysv fallback appeared
after ten crashes of --boot or --single postgres. Since --boot and --single
are rare in production use, systems used for PostgreSQL development are the
principal candidate to notice this symptom. Back-patch to 9.3 (all supported
versions). PostgreSQL 9.4 introduced dynamic shared memory, but 9.3 does
share the "SELECT random()" problem. Reviewed by Tom Lane and Kyotaro
HORIGUCHI. Discussion:
https://postgr.es/m/20180915221546.GA3159382@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/d18f6674bd60e923bcdbf0fd916685b0a250c60f

Joe Conway pushed:

- Document aclitem functions and operators. aclitem functions and operators
have been heretofore undocumented. Fix that. While at it, ensure the
non-operator aclitem functions have pg_description strings. Does not seem
worthwhile to back-patch. Author: Fabien Coelho, with pg_description from
John Naylor, and significant refactoring and editorialization by me. Reviewed
by: Tom Lane Discussion:
https://postgr.es/m/flat/alpine.DEB.2.21.1808010825490.18204%40lancre
https://git.postgresql.org/pg/commitdiff/c62dd80cdf149e2792b13c13777a539f5abb0370

Andrew Dunstan pushed:

- Fast default trigger and expand_tuple fixes. Ensure that triggers get
properly filled in tuples for the OLD value. Also fix the logic of detecting
missing null values. The previous logic failed to detect a missing null column
before the first missing column with a default. Fixing this has simplified the
logic a bit. Regression tests are added to test changes. This should ensure
better coverage of expand_tuple(). Original bug reports, and some code and
test scripts from Tomas Vondra Backpatch to release 11.
https://git.postgresql.org/pg/commitdiff/7636e5c60fea83a9f3cd2ad278c0819b98941c74

Andres Freund pushed:

- Make EXPLAIN output for JIT compilation more dense. A discussion about also
reporting JIT compilation overhead on workers brought unhappiness with the
verbosity of the current explain format to light. Make the text format more
dense, and restructure the structured output to mirror that more closely. As
we're re-jiggering the output format anyway: The denser format allows us to
report all flags for JIT compilation (now also reporting PGJIT_EXPR and
PGJIT_DEFORM), and report the total time in addition to the individual times.
Per complaint from Tom Lane. Author: Andres Freund Discussion:
https://postgr.es/m/27812.1537221015@sss.pgh.pa.us Backpatch: 11-, where JIT
compilation was introduced
https://git.postgresql.org/pg/commitdiff/52050ad8ebec8d831902f587314aa4f6aaa6d2c5

- auto_explain: Include JIT information if applicable. Due to my (Andres')
omission auto_explain did not include information about JIT compilation. Fix
that. Author: Lukas Fittl Discussion:
https://postgr.es/m/CAP53PkzgSyoTCau0-5FNaM484B=uO8nLzma7L1ncWLb1=oVJQA@mail.gmail.com
Backpatch: 11-, where JIT compilation was introduced
https://git.postgresql.org/pg/commitdiff/b076eb7669d7279d0f446305c2e12dffd6bc3347

- Collect JIT instrumentation from workers. Previously, when using parallel
query, EXPLAIN (ANALYZE)'s JIT compilation timings did not include the
overhead from doing so on the workers. Fix that. We do so by simply
aggregating the cost of doing JIT compilation on workers and the leader
together. Arguably that's not quite accurate, because the total time spend
doing so is spent in parallel - but it's hard to do much better. For
additional detail, when VERBOSE is specified, the stats for workers are
displayed separately. Author: Amit Khandekar and Andres Freund Discussion:
https://postgr.es/m/CAJ3gD9eLrz51RK_gTkod+71iDcjpB_N8eC6vU2AW-VicsAERpQ@mail.gmail.com
Backpatch: 11-
https://git.postgresql.org/pg/commitdiff/33001fd7a7072d483272115a9376478fdc007fb9

- Change TupleTableSlot->tts_nvalid to type AttrNumber. Previously it was an
int / 4 bytes. The maximum number of attributes in a tuple is restricted by
the maximum value Var->varattno, which is an AttrNumber/int16. Hence use the
same data type for TupleTableSlot->tts_nvalid. Author: Ashutosh Bapat
Discussion:
https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/a598708ffa8eb72a22eeee4e6f30bc26e4984acd

- Remove function list from prologue of execTuples.c. That section is never in
sync with the actual routines available and their functionality. Author:
Ashutosh Bapat Discussion:
https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/bbdfbb9154fccf5b58ecbbdf4e8989e2fed206f8

- Split ExecStoreTuple into ExecStoreHeapTuple and ExecStoreBufferHeapTuple.
Upcoming changes introduce further types of tuple table slots, in preparation
of making table storage pluggable. New storage methods will have different
representation of tuples, therefore the slot accessor should refer explicitly
to heap tuples. Instead of just renaming the functions, split it into one
function that accepts heap tuples not residing in buffers, and one accepting
ones in buffers. Previously one function was used for both, but that was a
bit awkward already, and splitting will allow us to represent slot types for
tuples in buffers and normal memory separately. This is split out from the
patch introducing abstract slots, as this largely consists out of mechanical
changes. Author: Ashutosh Bapat Reviewed-By: Andres Freund Discussion:
https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/29c94e03c7d05d2b29afa1de32795ce178531246

- Remove absolete function TupleDescGetSlot(). TupleDescGetSlot() was kept
around for backward compatibility for user-written SRFs. With the
TupleTableSlot abstraction work, that code will need to be version specific
anyway, so there's no point in keeping the function around any longer.
Author: Ashutosh Bapat Reviewed-By: Andres Freund Discussion:
https://postgr.es/m/20180220224318.gw4oe5jadhpmcdnm@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/10763358c3f0df48d2ae39b49b0c93be149cceab

- Clean up in the wake of TupleDescGetSlot() removal / 10763358c3f. The
previous commit wasn't careful enough to remove all traces of
TupleDescGetSlot(). Besides fixing the oversight of not removing
TupleDescGetSlot()'s declaration, this also removes FuncCallContext->slot.
That was documented to be for use in combination with TupleDescGetSlot(), a
cursory search over extensions finds no users, and there doesn't seem to be
convincing reasons to keep it around. If we later in the v12 release cycle
find users, we can re-consider this part of the commit. Reported-By: Michael
Paquier Discussion: https://postgr.es/m/20180926000413.GC1659@paquier.xyz
https://git.postgresql.org/pg/commitdiff/27e082b0c6e564facfbf54b56090fdcc4bf44cca

- Correct overflow handling in pgbench. This patch attempts, although it's
quite possible there are a few holes, to properly detect and reported signed
integer overflows in pgbench. Author: Fabien Coelho Reviewed-By: Andres
Freund Discussion:
https://postgr.es/m/20171212052943.k2hlckfkeft3eiio@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/92a0342a90b38b0b007f079d33286f9aefabfe40

Michaël Paquier pushed:

- Revoke pg_stat_statements_reset() permissions. Commit 25fff40 has granted
execute permission of the function pg_stat_statements_reset() to default role
"pg_read_all_stats", but this role is meant to read statistics, and not to
reset them. The permissions on this function are revoked from
"pg_read_all_stats". The version of pg_stat_statements is bumped up in
consequence. Author: Haribabu Kommi Reviewed-by: Michael Paquier, Amit Kapila
Discussion:
https://postgr.es/m/CAJrrPGf5fCnKqXObpwGN9nMyD--tzOf-7LFCJiz59Z1wJ5qj9A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/edb9797660541b217d23ae7c02b96b496d34fec4

- Ignore publication tables when --no-publications is used. 96e1cb4 has added
support for --no-publications in pg_dump, pg_dumpall and pg_restore, but
forgot the fact that publication tables also need to be ignored when this
option is used. Author: Gilles Darold Reviewed-by: Michael Paquier
Discussion:
https://postgr.es/m/3f48e812-b0fa-388e-2043-9a176bdee27e@dalibo.com
Backpatch-through: 10, where publications have been added.
https://git.postgresql.org/pg/commitdiff/08c9917e24683e36dca35201723e47cdc3fa62db

- Rework activation of commit timestamps during recovery. The activation and
deactivation of commit timestamp tracking has not been handled consistently
for a primary or standbys at recovery. The facility can be activated at three
different moments of recovery: - The beginning, where a primary would use the
GUC value for the decision-making, and where a standby relies on the contents
of the control file. - When replaying a XLOG_PARAMETER_CHANGE record at redo.
- The end, where both primary and standby rely on the GUC value. Using the
GUC value for a primary at the beginning of recovery causes problems with
commit timestamp access when doing crash recovery. Particularly, when
replaying transaction commits, it could be possible that an attempt to read
commit timestamps is done for a transaction which committed at a moment when
track_commit_timestamp was disabled. A test case is added to reproduce the
failure. The test works down to v11 as it takes advantage of transaction
commits within procedures. Reported-by: Hailong Li Author: Masahiko Sawasa,
Michael Paquier Reviewed-by: Kyotaro Horiguchi Discussion:
https://postgr.es/m/11224478-a782-203b-1f17-e4797b39bdf0@qunar.com
Backpatch-through: 9.5, where commit timestamps have been introduced.
https://git.postgresql.org/pg/commitdiff/8d28bf500f6536e295e9c3d7b85cdfec1c4dc913

- Add basic regression tests for default monitoring roles. The following
default roles gain some coverage: - pg_read_all_stats - pg_read_all_settings
Author: Alexandra Ryzhevich Discussion:
https://postgr.es/m/CAOt4E5S5WJmDc9YpS1BfyAMQ5C1NEmiYynD6nUz42qVxphqkpA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/f535d5f0c13cb48d85565017e7d56f2075c59978

- Switch flags tracking pending interrupts to sig_atomic_t. Those previously
used bool, which should be safe on any modern platforms, however the C
standard is clear that it is better to use sig_atomic_t for variables
manipulated in signal handlers. This commit adds at the same time PGDLLIMPORT
to ClientConnectionLost. Author: Michael Paquier Reviewed-by: Tom Lane, Chris
Travers, Andres Freund Discussion:
https://postgr.es/m/20180925011311.GD1354@paquier.xyz
https://git.postgresql.org/pg/commitdiff/ba16aade337b16028d1e9e156d83417097c13817

- Fix WAL recycling on standbys depending on archive_mode. A restart point or a
checkpoint recycling WAL segments treats segments marked with neither ".done"
(archiving is done) or ".ready" (segment is ready to be archived) in
archive_status the same way for archive_mode being "on" or "always". While
for a primary this is fine, a standby running a restart point with
archive_mode = on would try to mark such a segment as ready for archiving,
which is something that will never happen except after the standby is
promoted. Note that this problem applies only to WAL segments coming from the
local pg_wal the first time archive recovery is run. Segments part of a
self-contained base backup are the most common case where this could happen,
however even in this case normally the .done markers would be most likely part
of the backup. Segments recovered from an archive are marked as .ready or
.done by the startup process, and segments finished streaming are marked as
such by the WAL receiver, so they are handled already. Reported-by: Haruka
Takatsuka Author: Michael Paquier Discussion:
https://postgr.es/m/15402-a453c90ed4cf88b2@postgresql.org Backpatch-through:
9.5, where archive_mode = always has been added.
https://git.postgresql.org/pg/commitdiff/78ea8b5daab9237fd42d7a8a836c1c451765499f

Thomas Munro pushed:

- Constify dsa_size_class_map and use a better type. Author: Mark G
Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/CAEeOP_Zy_FvVwcAU0UX9nkOhnoR5KN%3D0B6LWX_kv0ZuSc4wbGw%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/64171b32069adcce7f57840143eaca9fbe28ba7e

Álvaro Herrera pushed:

- Remove fmgr.h inclusion from partition.h. It's not needed anymore.
https://git.postgresql.org/pg/commitdiff/62e533d3f129b59794c256eabad44011c9d2b951

- Remove obsolete comment. The documented shortcoming was actually fixed in
4c728f3829 so the comment is not true anymore.
https://git.postgresql.org/pg/commitdiff/5913b9bbf351b421141b300e37752e9ab8d85163

Tomáš Vondra:

- Improve test coverage of geometric types. This commit significantly increases
test coverage of geo_ops.c, adding tests for various issues addressed by
2e2a392de3 (which went undetected for a long time, at least partially due to
not being covered). This also removes alternative results expecting -0 on
some platforms. Instead the functions are should return the same results
everywhere, transforming -0 to 0 if needed. The tests are added to
geometric.sql file, sorted by the left hand side of the operators. There are
many cross datatype operators, so this seems like the best solution. Author:
Emre Hasegeli Reviewed-by: Tomas Vondra Discussion:
https://www.postgresql.org/message-id/CAE2gYzxF7-5djV6-cEvqQu-fNsnt%3DEqbOURx7ZDg%2BVv6ZMTWbg%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/a3d2844852dc664718320b15cbc6d6bfa264e66e

- Fix problems in handling the line data type. According to the source history,
the internal format of line data type has changed, but various functions
working with it did were not updated and thus were producing wrong results.
This patch addresses various such issues, in particular: * Reject invalid
specification A=B=0 on receive * Reject same points on line_construct_pp() *
Fix perpendicular operator when negative values are involved * Avoid division
by zero on perpendicular operator * Fix intersection and distance operators
when neither A nor B are 1 * Return NULL for closest point when objects are
parallel * Check whether closest point of line segments is the intersection
point * Fix closest point of line segments being on the wrong segment Aside
from handling those issues, the patch also aims to make operators more
symmetric and less sen to precision loss. The EPSILON interferes with even
minor changes, but the least we can do is applying it to both sides of the
operators equally. Author: Emre Hasegeli Reviewed-by: Tomas Vondra
Discussion:
https://www.postgresql.org/message-id/CAE2gYzxF7-5djV6-cEvqQu-fNsnt%3DEqbOURx7ZDg%2BVv6ZMTWbg%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2e2a392de391a9f4ef4221fccbd00c43ba5c9b40

Peter Eisentraut pushed:

- Update dummy CREATE ASSERTION grammar. While we are probably still far away
from fully implementing assertions, all patch proposals appear to take issue
with the existing dummy grammar CREATE/DROP ASSERTION productions, so update
those a little bit. Rename the rule, use any_name instead of name, and remove
some unused code. Also remove the production for DROP ASSERTION, since that
would most likely be handled via the generic DROP support. extracted from a
patch by Joe Wildish
https://git.postgresql.org/pg/commitdiff/a49ceda6a044c2fc104b3d1397fe0bef8679d1aa

- Recurse to sequences on ownership change for all relkinds. When a table
ownership is changed, we must apply that also to any owned sequences.
(Otherwise, it would result in a situation that cannot be restored, because
linked sequences must have the same owner as the table.) But this was
previously only applied to regular tables and materialized views. But it
should also apply to at least foreign tables. This patch removes the relkind
check altogether, because it doesn't save very much and just introduces the
possibility of similar omissions. Bug: #15238 Reported-by: Christoph Berg
<christoph(dot)berg(at)credativ(dot)de>
https://git.postgresql.org/pg/commitdiff/0320ddaf3c08ea17d616549d0e7f45239592fc76

Alexander Korotkov pushed:

- Remove extra usage of BoxPGetDatum() macro. Author: Mark Dilger Discussion:
https://postgr.es/m/B2AEFCD0-836D-4654-9D59-3DF616E0A6F3%40gmail.com
https://git.postgresql.org/pg/commitdiff/0f6459589494a4b4ff6c707594f8d308b9da88f8

- Minor formatting cleanup for 2a6368343f.
https://git.postgresql.org/pg/commitdiff/4ec90f53f10141867d8b86f58d72990a13ff267b

Amit Kapila pushed:

- Fix assertion failure when updating full_page_writes for checkpointer. When
the checkpointer receives a SIGHUP signal to update its configuration, it may
need to update the shared memory for full_page_writes and need to write a WAL
record for it. Now, it is quite possible that the XLOG machinery has not been
initialized by that time and it will lead to assertion failure while doing
that. Fix is to allow the initialization of the XLOG machinery outside
critical section. This bug has been introduced by the commit 2c03216d83 which
added the XLOG machinery initialization in RecoveryInProgress code path.
Reported-by: Dilip Kumar Author: Dilip Kumar Reviewed-by: Michael Paquier and
Amit Kapila Backpatch-through: 9.5 Discussion:
https://postgr.es/m/CAFiTN-u4BA8KXcQUWDPNgaKAjDXC=C2whnzBM8TAcv=stckYUw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/a86bf6057edf7bc344f1ab6ba7824b561ed9068a

Stephen Frost pushed:

- Add application_name to connection authorized msg. The connection authorized
message has quite a bit of useful information in it, but didn't include the
application_name (when provided), so let's add that as it can be very useful.
Note that at the point where we're emitting the connection authorized message,
we haven't processed GUCs, so it's not possible to get this by using
log_line_prefix (which pulls from the GUC). There's also something to be said
for having this included in the connection authorized message and then not
needing to repeat it for every line, as having it in log_line_prefix would do.
The GUC cleans the application name to pure-ascii, so do that here too, but
pull out the logic for cleaning up a string into its own function in common
and re-use it from those places, and check_cluster_name which was doing the
same thing. Author: Don Seiler <don(at)seiler(dot)us> Discussion:
https://postgr.es/m/CAHJZqBB_Pxv8HRfoh%2BAB4KxSQQuPVvtYCzMg7woNR3r7dfmopw%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/8bddc864000f56d396621d4ad0f13e8e1872ddf5

== Pending Patches ==

Thomas Munro sent in two more revisions of a patch to enable SERIALIZABLE READ
ONLY DEFERRABLE for hot standbys.

Thomas Munro sent in another revision of a patch to implement a synchronous
replay mode for avoiding stale reads on hot standbys.

Andrey Borodin sent in another revision of a patch to add a GiST verification
function for amcheck.

Peter Eisentraut sent in a patch to clarify CREATE TABLESPACE documentation by
being more specific about when and how to create the directory and what
permissions it should have.

Elvis Pranskevichus sent in another revision of a patch to add and report a new
GUC: session_read_only. This GUC is reported via a ParameterStatus message.

Haribabu Kommi sent in a patch to revoke the pg_stat_statements_reset()
permissions from the pg_read_all_stats role, which shouldn't have had them in
the first place.

Haribabu Kommi sent in three more revisions of a patch to add parameters userid,
dbid and queryid to pg_stat_statements_reset().

Jim Nasby sent in a patch to remove is_wraparound and move the check for
anti-wraparound VACUUM from vacuum_rel() to lazy_vacuum_rel().

Pavan Deolasee sent in another revision of a patch to implement MERGE per
SQL:2016.

Thomas Munro sent in a patch to add DNS SRV support for LDAP authentication.

Thomas Munro sent in a patch to de-support and remove dsm_resize() and
dsm_remap().

Surafel Temesgen sent in another revision of a patch to add a PERCENT option to
FETCH FIRST.

Michaël Paquier sent in another revision of a patch to fix a bug in
track_commit_timestamp.

Dmitry Dolgov and Michaël Paquier traded patches to fix a bug that manifested as
a segfault when creating partition with a primary key and sql_drop trigger
exists.

Sergei Kornilov sent in another revision of a patch to refactor the
recovery.conf API.

David Rowley sent in another revision of a patch to calculate total_table_pages
after set_base_rel_sizes.

Richard Guo sent in another revision of a patch to implement predicate
propagation for non-equivalence clauses.

Haozhou Wang sent in another revision of a patch to implement disk quotas.

Amul Sul sent in a patch to add a 64-bit hash function for the hstore and citext
data types.

Fabien COELHO sent in another revision of a patch to add a pseudo-random
permutation function to pgbench.

Liudmila Mantrova sent in another revision of a patch to clarify the handling of
letters and digits in to_date() and to_timestamp().

Nikita Glukhov sent in another revision of a patch to implement kNN searches for
btree indexes.

Tom Lane and Daniel Gustafsson traded patches to add support for Secure
Transport SSL library on macOS as an alternative to OpenSSL.

Michael Banck sent in two more revisions of a patch to add online verification
of page checksums.

Tom Lane sent in another revision of a patch to speed up snprintf.

Michaël Paquier sent in a patch to refactor vacuum open.

Kyotaro HORIGUCHI sent in another revision of a patch to implement a
shared-memory based stats collector.

Thomas Munro sent in another revision of a patch to Use pread()/pwrite() instead
of lseek() + read()/write() in systems that have the former.

Nikita Glukhov sent in a patch to fix a memory leak in PLyObject_ToJsonbValue().

David Rowley sent in a patch to fix some incorrect comments and an outdated
README for run-time pruning.

Jimmy Yih sent in a patch to get a more consistent view definition when a UNION
subquery contains undecorated constants.

Peter Eisentraut sent in another revision of a patch to enable using file
cloning in pg_upgrade where available.

David Rowley and Amit Kapila traded patches to avoid locking range table
relations in the executor, remove some useless fields from planner nodes, prune
PlanRowMark of relations that are pruned from a plan, and revise executor range
table relation opening/closing.

Haribabu Kommi sent in another revision of a patch to move declaration of the
synchronize_seqscans GUC from src/backend/access/heap/heapam.c to
src/backend/access/table/tableam.c, add a check_default_table_access_method hook
to verify the access method, and validate the index scan hook addition.

Masahiko Sawada sent in a patch to fix how autovacuum is logged.

Edmund Horner sent in two more revisions of a patch to improve TID scanning by
adding support for range quals to estimating the cost of the path.

Haribabu Kommi sent in another revision of a patch to allow target_session_attrs
in libpq to accept a prefer-read option.

Thomas Munro sent in another revision of a patch to fix some corner case
problems with fsync.

Amit Khandekar sent in another revision of a patch to allocate a dedicated slot
for each partitions that requires it and slotify partition tuple conversion.

Peter Eisentraut sent in a patch to advance the transaction timestamp in
intra-procedure transactions.

David Rowley sent in two more revisions of a patch to fix an incorrect setting
of heap insert options for partitioned tables.

Michael Banck sent in another revision of a patch to add an option for progress
reporting to pg_verify_checksums.

Thomas Munro sent in two more revisions of a patch to add kqueue(2) support for
WaitEventSet.

Michaël Paquier sent in two more revisions of a patch to use durable_unlink for
.ready and .done files for WAL segment removal.

Andres Freund sent in another revision of a patch to remove abstime and things
that depend on it.

David Hedberg sent in a patch to add pipe support to pg_dump and pg_restore.

Pavel Stěhule sent in two more revisions of a patch to implement schema
variables.

Marco Atzeri sent in a patch to fix the Cygwin linking rules.

Andres Freund sent in a patch to remove WITH OIDs support and convert catalog
table oids to explicit oid columns.

Amit Kapila sent in another revision of a patch to fix datum serialization.

Dmitry Dolgov sent in another revision of a patch to add generic type
subscripting and use same for arrays and JSONB.

John Naylor sent in a patch to sync the ECPG scanner with core.

Tom Lane sent in two revisions of a patch to make executor relation handling
better by ensuring that the caller holds a lock *at least as strong* as the one
being recorded in the range table entry.

Tom Lane sent in a patch to fix a problem where relations are being opened
without any lock whatever.

Christoph Moench-Tegeder sent in a patch to implement pg_ls_archive_status().