== PostgreSQL Weekly News - February 17, 2019 ==

== PostgreSQL Weekly News - February 17, 2019 ==

PostgreSQL bug fix releases 11.2, 10.7, 9.6.12, 9.5.16, 9.4.21, and 9.3.26 released.
Upgrade as soon as possible.
https://www.postgresql.org/about/news/1920/

PostgresLondon 2019 will be July 2-3, 2019 with an optional training day on
July 1. The CfP is open at https://goo.gl/forms/hsvZKAmq0c96XQ4l2 through March
15, 2019.
http://postgreslondon.org

== PostgreSQL Product News ==

pgBadger v10.3, a PostgreSQL log analyzer and graph tool written in
Perl, released.
https://github.com/darold/pgbadger/releases

pgAdmin4 4.2, a web- and native GUI control center for PostgreSQL, released.
https://www.pgadmin.org/docs/pgadmin4/dev/release_notes_4_2.html

== PostgreSQL Jobs for February ==

http://archives.postgresql.org/pgsql-jobs/2019-02/

== PostgreSQL Local ==

PostgreSQL(at)SCaLE is a two day, two track event which takes place on
March 7-8, 2019, at Pasadena Convention Center, as part of SCaLE 17X.
https://www.socallinuxexpo.org/scale/17x/postgresscale

pgDay Paris 2019 will be held in Paris, France on March 12, 2019
at 199bis rue Saint-Martin.
http://2019.pgday.paris/

Nordic PGDay 2019 will be held in Copenhagen, Denmark, at the
Copenhagen Marriott Hotel, on March 19, 2019.
https://2019.nordicpgday.org/

PGConf APAC 2019 will be held in Singapore March 19-21, 2019.
http://2019.pgconfapac.org/

The German-speaking PostgreSQL Conference 2019 will take place on May 10, 2019
in Leipzig. The CfP is open until February 26, 2019 at http://2019.pgconf.de/cfp
http://2019.pgconf.de/

PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy.
https://2019.pgday.it/en/

PGCon 2019 will take place in Ottawa on May 28-31, 2019.
https://www.pgcon.org/2019

Swiss PGDay 2019 will take place in Rapperswil (near Zurich) on June 28, 2019.
The CfP is open through April 18, 2019, and registration is open.
http://www.pgday.ch/2019/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Add per-test-script runtime display to pg_regress. It seems useful to have
this information available, so that it's easier to tell when a test script is
taking a disproportionate amount of time. Discussion:
https://postgr.es/m/16646.1549770618@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/72d71e03563b6c295b257040e324793a30162042

- Fix indexable-row-comparison logic to account for covering indexes. indxpath.c
needs a good deal more attention for covering indexes than it's gotten. But
so far as I can tell, the only really awful breakage is in
expand_indexqual_rowcompare (nee adjust_rowcompare_for_index), which was only
half fixed in c266ed31a. The other problems aren't bad enough to take the
risk of a just-before-wrap fix. The problem here is that if the leading
column of a row comparison matches an index (allowing this code to be
reached), and some later column doesn't match the index, it'll nonetheless
believe that that column matches the first included index column. Typically
that'll lead to an error like "operator M is not a member of opfamily N" as a
result of fetching a garbage opfamily OID. But with enough bad luck, maybe a
broken plan would be generated. Discussion:
https://postgr.es/m/25526.1549847928@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/6bdc3005b54fc9c6ef27cd4e64edd548315f57ba

- Redesign the partition dependency mechanism. The original setup for
dependencies of partitioned objects had serious problems: 1. It did not
verify that a drop cascading to a partition-child object also cascaded to at
least one of the object's partition parents. Now, normally a child object
would share all its dependencies with one or another parent (e.g. a child
index's opclass dependencies would be shared with the parent index), so that
this oversight is usually harmless. But if some dependency failed to fit this
pattern, the child could be dropped while all its parents remain, creating a
logically broken situation. (It's easy to construct artificial cases that
break it, such as attaching an unrelated extension dependency to the child
object and then dropping the extension. I'm not sure if any less-artificial
cases exist.) 2. Management of partition dependencies during ATTACH/DETACH
PARTITION was complicated and buggy; for example, after detaching a partition
table it was possible to create cases where a formerly-child index should be
dropped and was not, because the correct set of dependencies had not been
reconstructed. Less seriously, because multiple partition relationships were
represented identically in pg_depend, there was an order-of-traversal
dependency on which partition parent was cited in error messages. We also had
some pre-existing order-of-traversal hazards for error messages related to
internal and extension dependencies. This is cosmetic to users but causes
testing problems. To fix #1, add a check at the end of the partition tree
traversal to ensure that at least one partition parent got deleted. To fix
#2, establish a new policy that partition dependencies are in addition to, not
instead of, a child object's usual dependencies; in this way ATTACH/DETACH
PARTITION need not cope with adding or removing the usual dependencies. To
fix the cosmetic problem, distinguish between primary and secondary partition
dependency entries in pg_depend, by giving them different deptypes. (They
behave identically except for having different priorities for being cited in
error messages.) This means that the former 'I' dependency type is replaced
with new 'P' and 'S' types. This also fixes a longstanding bug that after
handling an internal dependency by recursing to the owning object,
findDependentObjects did not verify that the current target was now scheduled
for deletion, and did not apply the current recursion level's objflags to it.
Perhaps that should be back-patched; but in the back branches it would only
matter if some concurrent transaction had removed the internal-linkage
pg_depend entry before the recursive call found it, or the recursive call
somehow failed to find it, both of which seem unlikely. Catversion bump
because the contents of pg_depend change for partitioning relationships.
Patch HEAD only. It's annoying that we're not fixing #2 in v11, but there
seems no practical way to do so given that the problem is exactly a poor
choice of what entries to put in pg_depend. We can't really fix that while
staying compatible with what's in pg_depend in existing v11 installations.
Discussion:
https://postgr.es/m/CAH2-Wzkypv1R+teZrr71U23J578NnTBt2X8+Y=Odr4pOdW1rXg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/1d92a0c9f7dd547ad14cd8a3974289c5ec7f04ce

- Allow extensions to generate lossy index conditions. For a long time,
indxpath.c has had the ability to extract derived (lossy) index conditions
from certain operators such as LIKE. For just as long, it's been obvious that
we really ought to make that capability available to extensions. This commit
finally accomplishes that, by adding another API for planner support functions
that lets them create derived index conditions for their functions. As proof
of concept, the hardwired "special index operator" code formerly present in
indxpath.c is pushed out to planner support functions attached to LIKE and
other relevant operators. A weak spot in this design is that an extension
needs to know OIDs for the operators, datatypes, and opfamilies involved in
the transformation it wants to make. The core-code prototypes use hard-wired
OID references but extensions don't have that option for their own operators
etc. It's usually possible to look up the required info, but that may be slow
and inconvenient. However, improving that situation is a separate task. I
want to do some additional refactorization around selfuncs.c, but that also
seems like a separate task. Discussion:
https://postgr.es/m/15193.1548028093@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/74dfe58a5927b22c744b29534e67bfdd203ac028

- Fix header inclusion issue. partprune.h failed to compile by itself; needs to
include partdefs.h. I think I must've broken this in fa2cf164a, though I'd
swear I ran the appropriate tests when removing #includes. Anyway, it's very
sensible for this file to include partdefs.h, so let's just do that. Per
cpluspluscheck.
https://git.postgresql.org/pg/commitdiff/b07c695d9c34cccfa0138ca7f4c76547a24c74e1

- Fix erroneous error reports in snapbuild.c. It's pretty unhelpful to report
the wrong file name in a complaint about syscall failure, but
SnapBuildSerialize managed to do that twice in a span of 50 lines. Also fix
half a dozen missing or poorly-chosen errcode assignments; that's mostly
cosmetic, but still wrong. Noted while studying recent failures on buildfarm
member nightjar. I'm not sure whether those reports are actually giving the
wrong filename, because there are two places here with identically spelled
error messages. The other one is specifically coded not to report ENOENT, but
if it's this one, how could we be getting ENOENT from open() with O_CREAT?
Need to sit back and await results. However, these ereports are clearly
broken from birth, so back-patch.
https://git.postgresql.org/pg/commitdiff/232a8e233f21eb9a214c551d5a6af3d324915b4e

- Clean up planner confusion between ncolumns and nkeycolumns. We're only going
to consider key columns when creating indexquals, so there is no point in
having the outer loops in indxpath.c iterate further than nkeycolumns. Doing
so in match_pathkeys_to_index() is actually wrong, and would have caused
crashes by now, except that we have no index AMs supporting both
amcanorderbyop and amcaninclude. It's also wrong in
relation_has_unique_index_for(). The effect there is to fail to prove
uniqueness even when the index does prove it, if there are extra columns.
Also future-proof examine_variable() for the day when extra columns can be
expressions, and fix what's either a thinko or just an oversight in
btcostestimate(): we should consider the number of key columns, not the total,
when deciding whether to derate correlation. None of these things seemed
important enough to risk changing in a just-before-wrap patch, but since we're
past the release wrap window, time to fix 'em. Discussion:
https://postgr.es/m/25526.1549847928@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/75c46149fc2215b046aa56caaf59f8125f0e6062

- Move pattern selectivity code from selfuncs.c to like_support.c. While at it,
refactor patternsel() a bit so that it can be used from the LIKE/regex planner
support functions as well. This makes the planner able to deal equally well
with either operator or function syntax for these operations. I'm not excited
about that as a feature in itself, but it provides a nice model for extensions
to follow if they want such behavior for their operations. This change
localizes the use of pattern_fixed_prefix() and make_greater_string() so that
they no longer need be exported. (We might get pushback from extensions about
that, perhaps, in which case I'd be inclined to re-export them in a new header
file like_support.h.) This reduces the bulk of selfuncs.c a fair amount,
removing ~1370 lines or about one-sixth of that file; it's still too big, but
this is progress. Discussion:
https://postgr.es/m/24537.1550093915@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/49fa99e54ec0ded180b52a4a14e543702d53e8c9

- Simplify the planner's new representation of indexable clauses a little. In
commit 1a8d5afb0, I thought it'd be a good idea to define
IndexClause.indexquals as NIL in the most common case where the given clause
(IndexClause.rinfo) is usable exactly as-is. It'd be more consistent to
define the indexquals in that case as being a one-element list containing
IndexClause.rinfo, but I thought saving the palloc overhead for making such a
list would be worthwhile. In hindsight, that was a great example of
"premature optimization is the root of all evil": it's complicated everyplace
that needs to deal with the indexquals, requiring duplicative code to handle
both the simple case and the not-simple case. I'd initially found that
tolerable but it's getting less so as I mop up some areas that I'd not touched
in 1a8d5afb0. In any case, two more pallocs during a planner run are surely
at the noise level (a conclusion confirmed by a bit of microbenchmarking). So
let's change this decision before it becomes set in stone, and insist that
IndexClause.indexquals always be a valid list of the actual index quals for
the clause. Discussion: https://postgr.es/m/24586.1550106354@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/8fd3fdd85a3e22f04b2cb0949450f63cb48952cd

- Refactor index cost estimation functions in view of IndexClause changes. Get
rid of deconstruct_indexquals() in favor of just iterating over the
IndexClause list directly. The extra services that that function used to
provide, such as hiding clause commutation and associating the right index
column with each clause, are no longer useful given the new data structure.
I'd originally thought that it'd provide a useful amount of abstraction by
freeing callers from paying attention to the exact clause type of each
indexqual, but that hope proves to have been vain, because few callers can
ignore the semantic differences between different clause types. Indeed,
removing it results in a net code savings, and probably some cycles shaved by
not having to build an extra list-of-structs data structure. Also, export a
few formerly-static support functions, with the goal of allowing extension AMs
to write functionality equivalent to genericcostestimate() without pointless
code duplication. Discussion:
https://postgr.es/m/24586.1550106354@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/e89f14e2bb9f7c392c4c85a53ab5a13ea2aed83d

- Make use of compiler builtins and/or assembly for CLZ, CTZ, POPCNT. Test for
the compiler builtins __builtin_clz, __builtin_ctz, and __builtin_popcount,
and make use of these in preference to handwritten C code if they're
available. Create src/port infrastructure for "leftmost one", "rightmost
one", and "popcount" so as to centralize these decisions. On x86_64,
__builtin_popcount generally won't make use of the POPCNT opcode because
that's not universally supported yet. Provide code that checks CPUID and then
calls POPCNT via asm() if available. This requires indirecting through a
function pointer, which is an annoying amount of overhead for a
one-instruction operation, but it's probably not worth working harder than
this for our current use-cases. I'm not sure we've found all the existing
places that could profit from this new infrastructure; but we at least touched
all the ones that used copied-and-pasted versions of the bitmapset.c code, and
got rid of multiple copies of the associated constant arrays. While at it,
replace c-compiler.m4's one-per-builtin-function macros with a single one that
can handle all the cases we need to worry about so far. Also, because I'm
paranoid, make those checks into AC_LINK checks rather than just AC_COMPILE;
the former coding failed to verify that libgcc has support for the builtin, in
cases where it's not inline code. David Rowley, Thomas Munro, Alvaro Herrera,
Tom Lane Discussion:
https://postgr.es/m/CAKJS1f9WTAGG1tPeJnD18hiQW5gAk59fQ6WK-vfdAKEHyRg2RA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/02a6a54ecd6632f974b1b4eebfb2373363431084

- Allow user control of CTE materialization, and change the default behavior.
Historically we've always materialized the full output of a CTE query,
treating WITH as an optimization fence (so that, for example, restrictions
from the outer query cannot be pushed into it). This is appropriate when the
CTE query is INSERT/UPDATE/DELETE, or is recursive; but when the CTE query is
non-recursive and side-effect-free, there's no hazard of changing the query
results by pushing restrictions down. Another argument for materialization is
that it can avoid duplicate computation of an expensive WITH query --- but
that only applies if the WITH query is called more than once in the outer
query. Even then it could still be a net loss, if each call has restrictions
that would allow just a small part of the WITH query to be computed. Hence,
let's change the behavior for WITH queries that are non-recursive and
side-effect-free. By default, we will inline them into the outer query
(removing the optimization fence) if they are called just once. If they are
called more than once, we will keep the old behavior by default, but the user
can override this and force inlining by specifying NOT MATERIALIZED. Lastly,
the user can force the old behavior by specifying MATERIALIZED; this would
mainly be useful when the query had deliberately been employing WITH as an
optimization fence to prevent a poor choice of plan. Andreas Karlsson, Andrew
Gierth, David Fetter Discussion:
https://postgr.es/m/87sh48ffhb.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/608b167f9f9c4553c35bb1ec0eab9ddae643989b

- Fix CREATE VIEW to allow zero-column views. We should logically have allowed
this case when we allowed zero-column tables, but it was overlooked. Although
this might be thought a feature addition, it's really a bug fix, because it
was possible to create a zero-column view via the convert-table-to-view code
path, and then you'd have a situation where dump/reload would fail. Hence,
back-patch to all supported branches. Arrange the added test cases to provide
coverage of the related pg_dump code paths (since these views will be dumped
and reloaded during the pg_upgrade regression test). I also made them test
the case where pg_dump has to postpone the view rule into post-data, which
disturbingly had no regression coverage before. Report and patch by Ashutosh
Sharma (test case by me) Discussion:
https://postgr.es/m/CAE9k0PkmHdeSaeZt2ujnb_cKucmK3sDDceDzw7+d5UZoNJPYOg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/a32ca7883629f6b1fbbf0bd2e2aa11ec86edb6b3

Peter Eisentraut pushed:

- Remove unused macro. Last use was removed in
2c66f9924c1162bfba27c77004ccf42fb6ea188d.
https://git.postgresql.org/pg/commitdiff/78b0cac74dabf879f769ff02a0df83ed90f2d36c

- Adjust gratuitously different error message wording.
https://git.postgresql.org/pg/commitdiff/256fc004afb1eedba10232349c5149c8f41d06a1

- Remove useless casts. Some of these were uselessly casting away "const", some
were just nearby, but they where all unnecessary anyway. Discussion:
https://www.postgresql.org/message-id/flat/53a28052-f9f3-1808-fed9-460fd43035ab%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/cf40dc65b676c8df1ee12f060b40f0e37a183e04

- More unconstify use. Replace casts whose only purpose is to cast away const
with the unconstify() macro. Discussion:
https://www.postgresql.org/message-id/flat/53a28052-f9f3-1808-fed9-460fd43035ab%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/37d9916020286caec810f4de61fbd0de3568454d

- Resolve one unconstify use. A small API change makes it unnecessary.
Reported-by: Mark Dilger <hornschnorter(at)gmail(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/53a28052-f9f3-1808-fed9-460fd43035ab%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/4b3b07fd5d60cffc95cabf34246d71d15b0c8302

- Get rid of another unconstify through API changes. This also makes the code in
read_client_first_message() more similar to read_client_final_message().
Reported-by: Mark Dilger <hornschnorter(at)gmail(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/53a28052-f9f3-1808-fed9-460fd43035ab%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/86eea78694ce0d95e74cc82cc29c096d66a9accd

- Use standard diff separator for regression.diffs. Instead of ======..., use
the standard separator for a multi-file diff, which is, per POSIX, "diff
%s %s %s\n", <diff_options>, <filename1>, <filename2> This makes
regression.diffs behave more like a proper diff file, for use with other
tools. And it shows the diff options used, for clarity. Discussion:
https://www.postgresql.org/message-id/70440c81-37bb-76dd-e48b-b5a9550d5613@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/8f27a14b1bd3d906144356ce19e33a2fd0095141

- doc: Update README.links. The guideline to "not use text with <ulink> so the
URL appears in printed output" is obsolete, since the URL appears as a
footnote in printed output in that case. Reported-by: Chapman Flack
<chap(at)anastigmatix(dot)net> Discussion:
https://www.postgresql.org/message-id/5C4B1F2F.2000106@anastigmatix.net
https://git.postgresql.org/pg/commitdiff/b060e6c1f5b4609718468a0b6562dd03db26ea46

Álvaro Herrera pushed:

- Fix misleading PG_RE_THROW commentary. The old verbiage indicated that
PG_RE_THROW is optional, which is not really true. This has confused many
people, so it seems worth fixing. Discussion:
https://postgr.es/m/20190206160958.GA22304@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/c603b392c32513982439bc466312d3a6b697d53e

- Use Getopt::Long for catalog scripts. Replace hand-rolled option parsing with
the Getopt module. This is shorter and easier to read. In passing, make some
cosmetic adjustments for consistency. Author: John Naylor Reviewed-by: David
Fetter Discussion:
https://postgr.es/m/CACPNZCvRjepXh5b2N50njN+rO_2Nzcf=jhMkKX7=79XWUKJyKA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/fe33a196ded8565d0fd8367e816d695b840e40cb

- Blind attempt at fixing Windows build. Broken since fe33a196de.
https://git.postgresql.org/pg/commitdiff/d357a16997a2e9dce0f56299c739b2b2584c4026

- Relax overly strict assertion. Ever since its birth,
ReorderBufferBuildTupleCidHash() has contained an assertion that a catalog
tuple cannot change Cmax after acquiring one. But that's wrong: if a
subtransaction executes DDL that affects that catalog tuple, and later aborts
and another DDL affects the same tuple, it will change Cmax. Relax the
assertion to merely verify that the Cmax remains valid and monotonically
increasing, instead. Add a test that tickles the relevant code. Diagnosed
by, and initial patch submitted by: Arseny Sher Co-authored-by: Arseny Sher
Discussion: https://postgr.es/m/874l9p8hyw.fsf@ars-thinkpad
https://git.postgresql.org/pg/commitdiff/8c67d29fd51c0381d8bce41d35d7726725924616

- Add basic support for using the POPCNT and SSE4.2s LZCNT opcodes. These
opcodes have been around in the AMD world since 2007, and 2008 in the case of
intel. They're supported in GCC and Clang via some __builtin macros. The
opcodes may be unavailable during runtime, in which case we fall back on a
C-based implementation of the code. In order to get the POPCNT instruction we
must pass the -mpopcnt option to the compiler. We do this only for the
pg_bitutils.c file. David Rowley (with fragments taken from a patch by Thomas
Munro) Discussion:
https://postgr.es/m/CAKJS1f9WTAGG1tPeJnD18hiQW5gAk59fQ6WK-vfdAKEHyRg2RA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/711bab1e4d19b5c9967328315a542d93386b1ac5

- Add -mpopcnt to all compiles of pg_bitutils.c. The way this makefile works, we
need to specify it three times.
https://git.postgresql.org/pg/commitdiff/d0b4663c23b7a6ae6f489c4d7a2f58f879914959

- Fix portability issues in pg_bitutils. We were using uint64 function arguments
as "long int" arguments to compiler builtins, which fails on machines where
long ints are 32 bits: the upper half of the uint64 was being ignored. Fix by
using the "ll" builtin variants instead, which on those machines take 64 bit
arguments. Also, remove configure tests for __builtin_popcountl() (as well as
"long" variants for ctz and clz): the theory here is that any compiler version
will provide all widths or none, so one test suffices. Were this theory to be
wrong, we'd have to add tests for __builtin_popcountll() and friends, which
would be tedious. Per failures in buildfarm member lapwing and ensuing
discussion.
https://git.postgresql.org/pg/commitdiff/109de05cbb034b032cd60f50708716c8ff0afdf2

- Fix compiler builtin usage in new pg_bitutils.c. Split out these new functions
in three parts: one in a new file that uses the compiler builtin and gets
compiled with the -mpopcnt compiler option if it exists; another one that uses
the compiler builtin but not the compiler option; and finally the fallback
with open-coded algorithms. Split out the configure logic: in the original
commit, it was selecting to use the -mpopcnt compiler switch together with
deciding whether to use the compiler builtin, but those two things are really
separate. Split them out. Also, expose whether the builtin exists to
Makefile.global, so that src/port's Makefile can decide whether to compile the
hw-optimized file. Remove CPUID test for CTZ/CLZ. Make
pg_{right,left}most_ones use either the compiler intrinsic or open-coded algo;
trying to use the HW-optimized version is a waste of time. Make them static
inline functions. Discussion:
https://postgr.es/m/20190213221719.GA15976@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/fc6c72747ae6db6909fcd7d1adbc3d923ec1fffa

- Revert attempts to use POPCNT etc instructions. This reverts commits
fc6c72747ae6, 109de05cbb03, d0b4663c23b7 and 711bab1e4d19. Somebody will have
to try harder before submitting this patch again. I've spent entirely too much
time on it already, and the #ifdef maze yet to be written in order for it to
build at all got on my nerves. The amount of work needed to get a
platform-specific performance improvement that's barely above the noise level
is not worth it.
https://git.postgresql.org/pg/commitdiff/457aef0f1fd365c68fab3fa2ca3ae48c5bd230c6

Michaël Paquier pushed:

- Move max_wal_senders out of max_connections for connection slot handling.
Since its introduction, max_wal_senders is counted as part of max_connections
when it comes to define how many connection slots can be used for replication
connections with a WAL sender context. This can lead to confusion for some
users, as it could be possible to block a base backup or replication from
happening because other backend sessions are already taken for other purposes
by an application, and superuser-only connection slots are not a correct
solution to handle that case. This commit makes max_wal_senders independent
of max_connections for its handling of PGPROC entries in ProcGlobal, meaning
that connection slots for WAL senders are handled using their own free queue,
like autovacuum workers and bgworkers. One compatibility issue that this
change creates is that a standby now requires to have a value of
max_wal_senders at least equal to its primary. So, if a standby created
enforces the value of max_wal_senders to be lower than that, then this could
break failovers. Normally this should not be an issue though, as any settings
of a standby are inherited from its primary as postgresql.conf gets normally
copied as part of a base backup, so parameters would be consistent. Author:
Alexander Kukushkin Reviewed-by: Kyotaro Horiguchi, Petr Jelínek, Masahiko
Sawada, Oleksii Kliukin Discussion:
https://postgr.es/m/CAFh8B=nBzHQeYAu0b8fjK-AF1X4+_p6GRtwG+cCgs6Vci2uRuQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ea92368cd1da1e290f9ab8efb7f60cb7598fc310

- Clarify docs about limitations of constraint exclusion with partitions. The
current wording can confuse the reader about constraint exclusion being
available at query execution, but this only applies to partition pruning.
Reported-by: Shouyu Luo Author: David Rowley Reviewed-by: Chapman Flack, Amit
Langote Discussion: https://postgr.es/m/15629-2ef8b22e61f8333f@postgresql.org
https://git.postgresql.org/pg/commitdiff/ea05b221c2ff9d180f632ae90c806e984f15ed0d

- Fix description of WAL record XLOG_PARAMETER_CHANGE. max_wal_senders and
max_worker_processes got reversed in the output generated because of ea92368.
Reported-by: Kevin Hale Boyes Discussion:
https://postgr.es/m/CADAecHVAD4=26KAx4nj5DBvxqqvJkuwsy+riiiNhQqwnZg2K8Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/b7ec820559b68df446c01fe1497bd24e9091f559

- Fix comment related to calculation location of total_table_pages. As of commit
c6e4133, the calculation happens in make_one_rel() and not query_planner().
Author: Amit Langote Discussion:
https://postgr.es/m/c7a04a90-42e6-28a4-811a-a7e352831ba1@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/6ea95166a0f19ca0363b9c868e676b10365edec9

- Fix support for CREATE TABLE IF NOT EXISTS AS EXECUTE. The grammar IF NOT
EXISTS for CTAS is supported since 9.5 and documented as such, however the
case of using EXECUTE as query has never been covered as EXECUTE CTAS
statements and normal CTAS statements are parsed separately. Author: Andreas
Karlsson Discussion:
https://postgr.es/m/2ddcc188-e37c-a0be-32bf-a56b07c3559e@proxel.se
Backpatch-through: 9.5
https://git.postgresql.org/pg/commitdiff/331a613e9d363febfee8508e8de545fd84624758

Thomas Munro pushed:

- Fix rare dsa_allocate() failures due to freepage.c corruption. In a corner
case, a btree page was allocated during a clean-up operation that could cause
the tracking of the largest contiguous span of free space to get out of whack.
That was supposed to be prevented by the use of the "soft" flag to avoid
allocating internal pages during incidental clean-up work, but the flag was
ignored in the case where the FPM was promoted from singleton format to btree
format. Repair. Remove an obsolete comment in passing. Back-patch to 10,
where freepage.c arrived (as support for dsa.c). Author: Robert Haas
Diagnosed-by: Thomas Munro and Robert Haas Reported-by: Justin Pryzby, Rick
Otten, Sand Stone, Arne Roland and others Discussion:
https://postgr.es/m/CAMAYy4%2Bw3NTBM5JLWFi8twhWK4%3Dk_5L4nV5%2BbYDSPu8r4b97Zg%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7215efdc005e694ec93678a6203dbfc714d12809

- Fix race in dsm_attach() when handles are reused. DSM handle values can be
reused as soon as the underlying shared memory object has been destroyed.
That means that for a brief moment we might have two DSM slots with the same
handle. While trying to attach, if we encounter a slot with refcnt == 1,
meaning that it is currently being destroyed, we should continue our search in
case the same handle exists in another slot. The race manifested as a rare
"dsa_area could not attach to segment" error, and was more likely in 10 and 11
due to the lack of distinct seed for random() in parallel workers. It was
made very unlikely in in master by commit 197e4af9, and older releases don't
usually create new DSM segments in background workers so it was also unlikely
there. This fixes the root cause of bug report #15585, in which the error
could also sometimes result in a self-deadlock in the error path. It's not yet
clear if further changes are needed to avoid that failure mode. Back-patch to
9.4, where dsm.c arrived. Author: Thomas Munro Reported-by: Justin Pryzby,
Sergei Kornilov Discussion:
https://postgr.es/m/20190207014719.GJ29720@telsasoft.com Discussion:
https://postgr.es/m/15585-324ff6a93a18da46@postgresql.org
https://git.postgresql.org/pg/commitdiff/6c0fb941892519ad6b8873e99c4001404fb9a128

- Fix race in dsm_unpin_segment() when handles are reused. Teach
dsm_unpin_segment() to skip segments that are in the process of being
destroyed by another backend, when searching for a handle. Such a segment
cannot possibly be the one we are looking for, even if its handle matches.
Another slot might hold a recently created segment that has the same handle
value by coincidence, and we need to keep searching for that one. The bug
caused rare "cannot unpin a segment that is not pinned" errors on 10 and 11.
Similar to commit 6c0fb941 for dsm_attach(). Back-patch to 10, where
dsm_unpin_segment() landed. Author: Thomas Munro Reported-by: Justin Pryzby
Tested-by: Justin Pryzby (along with other recent DSA/DSM fixes) Discussion:
https://postgr.es/m/20190216023854.GF30291@telsasoft.com
https://git.postgresql.org/pg/commitdiff/0b55aaacec313c309e21f63d9ff5733c3f8ab813

Andrew Gierth pushed:

- Use strtof() and not strtod() for float4 input. Using strtod() creates a
double-rounding problem; the input decimal value is first rounded to the
nearest double; rounding that to the nearest float may then give an incorrect
result. An example is that 7.038531e-26 when input via strtod and then
rounded to float4 gives 0xAE43FEp-107 instead of the correct 0xAE43FDp-107.
Values output by earlier PG versions with extra_float_digits=3 should all be
read in with the same values as previously. However, values supplied by other
software using shortest representations could be mis-read. On platforms that
lack a strtof() entirely, we fall back to the old incorrect rounding behavior.
(As strtof() is required by C99, such platforms are considered of primarily
historical interest.) On VS2013, some workarounds are used to get correct
error handling. The regression tests now test for the correct input values,
so platforms that lack strtof() will need resultmap entries. An entry for
HP-UX 10 is included (more may be needed). Reviewed-By: Tom Lane Discussion:
https://postgr.es/m/871s5emitx.fsf@news-spur.riddles.org.uk Discussion:
https://postgr.es/m/87d0owlqpv.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/f397e08599a3c3c08b3af3b318c531db5882f57d

- Change floating-point output format for improved performance. Previously,
floating-point output was done by rounding to a specific decimal precision; by
default, to 6 or 15 decimal digits (losing information) or as requested using
extra_float_digits. Drivers that wanted exact float values, and applications
like pg_dump that must preserve values exactly, set extra_float_digits=3 (or
sometimes 2 for historical reasons, though this isn't enough for float4).
Unfortunately, decimal rounded output is slow enough to become a noticable
bottleneck when dealing with large result sets or COPY of large tables when
many floating-point values are involved. Floating-point output can be done
much faster when the output is not rounded to a specific decimal length, but
rather is chosen as the shortest decimal representation that is closer to the
original float value than to any other value representable in the same
precision. The recently published Ryu algorithm by Ulf Adams is both
relatively simple and remarkably fast. Accordingly, change
float4out/float8out to output shortest decimal representations if
extra_float_digits is greater than 0, and make that the new default.
Applications that need rounded output can set extra_float_digits back to 0 or
below, and take the resulting performance hit. We make one concession to
portability for systems with buggy floating-point input: we do not output
decimal values that fall exactly halfway between adjacent representable binary
values (which would rely on the reader doing round-to-nearest-even correctly).
This is known to be a problem at least for VS2013 on Windows. Our version of
the Ryu code originates from https://github.com/ulfjack/ryu/ at commit
c9c3fb1979, but with the following (significant) modifications: - Output
format is changed to use fixed-point notation for small exponents, as
printf would, and also to use lowercase 'e', a minimum of 2 exponent
digits, and a mandatory sign on the exponent, to keep the formatting as
close as possible to previous output. - The output of exact midpoint values
is disabled as noted above. - The integer fast-path code is changed somewhat
(since we have fixed-point output and the upstream did not). - Our
project style has been largely applied to the code with the exception of
C99 declaration-after-statement, which has been retained as an exception to
our present policy. - Most of upstream's debugging and conditionals are
removed, and we use our own configure tests to determine things like
uint128 availability. Changing the float output format obviously affects a
number of regression tests. This patch uses an explicit setting of
extra_float_digits=0 for test output that is not expected to be exactly
reproducible (e.g. due to numerical instability or differing algorithms for
transcendental functions). Conversions from floats to numeric are unchanged
by this patch. These may appear in index expressions and it is not yet clear
whether any change should be made, so that can be left for another day. This
patch assumes that the only supported floating point format is now IEEE
format, and the documentation is updated to reflect that. Code by me,
adapting the work of Ulf Adams and other contributors. References:
https://dl.acm.org/citation.cfm?id=3192369 Reviewed-by: Tom Lane, Andres
Freund, Donald Dong Discussion:
https://postgr.es/m/87r2el1bx6.fsf@news-spur.riddles.org.uk
https://git.postgresql.org/pg/commitdiff/02ddd499322ab6f2f0d58692955dc9633c2150fc

- Fix an overlooked UINT32_MAX. Replace with PG_UINT32_MAX. Per buildfarm
members dory and woodlouse.
https://git.postgresql.org/pg/commitdiff/754ca99314e9e1debe855b0462869ef6e58b7e7a

- More float test and portability fixes. Avoid assuming exact results in tstypes
test; some platforms vary. (per buildfarm members eulachon, danio, lapwing)
Avoid dubious usage (inherited from upstream) of bool parameters to
copy_special_str, to see if this fixes the mac/ppc failures (per buildfarm
members prariedog and locust). (Isolated test programs on a ppc mac don't seem
to show any other cause that would explain them.)
https://git.postgresql.org/pg/commitdiff/da6520be7f946f5f0f8fe46c34e303d1d36ee080

- Remove a stray subnormal value from float tests. We don't care to assume that
input of subnormal float values works, but a stray subnormal value from the
upstream Ryu regression test had been left in the test data by mistake. Remove
it. Per buildfarm member fulmar.
https://git.postgresql.org/pg/commitdiff/80c468b4a454881b56e1c73c6fedcb2978c5b415

- Cygwin and Mingw floating-point fixes. Deal with silent-underflow errors in
float4 for cygwin and mingw by using our strtof() wrapper; deal with
misrounding errors by adding them to the resultmap. Some slight reorganization
of declarations was done to avoid duplicating material between cygwin.h and
win32_port.h. While here, remove from the resultmap all references to
float8-small-is-zero; inspection of cygwin output suggests it's no longer
required there, and the freebsd/netbsd/openbsd entries should no longer be
necessary (these date back to c. 2000). This commit doesn't remove the file
itself nor the documentation references for it; that will happen in a
subsequent commit if all goes well.
https://git.postgresql.org/pg/commitdiff/72880ac182c8f3769f0be868f77166994282cb2a

- Fix previous MinGW fix. Definitions required for MinGW need to be outside #if
_MSC_VER. Oops.
https://git.postgresql.org/pg/commitdiff/79730e2a9bb1ce7837feddd16208ff2d9e490118

- Remove float8-small-is-zero regression test variant. Since this was also the
variant used as an example in the docs, update the docs to use
float4-misrounded-input as an example instead, since that is now the only
remaining variant file.
https://git.postgresql.org/pg/commitdiff/6ee89952d4e944b5c9c26181d418395b2f5d5fd8

Michael Meskes pushed:

- Add DECLARE STATEMENT support to ECPG. DECLARE STATEMENT is a statement that
lets users declare an identifier pointing at a connection. This identifier
will be used in other embedded dynamic SQL statement such as PREPARE, EXECUTE,
DECLARE CURSOR and so on. When connecting to a non-default connection, the AT
clause can be used in a DECLARE STATEMENT once and is no longer needed in
every dynamic SQL statement. This makes ECPG applications easier and more
efficient. Moreover, writing code without designating connection explicitly
improves portability. Authors: Ideriha-san ("Ideriha, Takeshi"
<ideriha(dot)takeshi(at)jp(dot)fujitsu(dot)com>) Kuroda-san ("Kuroda, Hayato"
<kuroda(dot)hayato(at)jp(dot)fujitsu(dot)com>) Discussion:
https://postgr.es/m4E72940DA2BF16479384A86D54D0988A565669DF(at)G01JPEXMBKW04
https://git.postgresql.org/pg/commitdiff/bd7c95f0c1a38becffceb3ea7234d57167f6d4bf

Noah Misch pushed:

- Import changes from IMath versions (1.3, 1.29]. Upstream fixed bugs over the
years, but none are confirmed to have affected pgcrypto. We're better off
naively tracking upstream than reactively maintaining a twelve-year-old
snapshot of upstream. Add a header comment describing the synchronization
procedure. Discard use of INVERT_COMPARE_RESULT(); the domain of the
comparisons in question is {-1,0,1}, controlled entirely by code in imath.c.
Andrew Gierth reviewed the Makefile change. Tom Lane reviewed the
synchronization procedure description. Discussion:
https://postgr.es/m/20190203035704.GA6226@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/48e24ba6b7fd3bfd156b51e8d768fd48df0d288b

- Fix PERMIT_DECLARATION_AFTER_STATEMENT initialization. The defect caused a
mere warning and only for gcc versions before 3.4.0.
https://git.postgresql.org/pg/commitdiff/d1299aabbd0b4ae860078691b628dc6b90698039

- In imath.h, replace stdint.h usage with c.h equivalents. As things stood,
buildfarm member dory failed. MSVC versions lacking stdint.h are unusable for
building PostgreSQL, but pg_config.h.win32 doesn't know that. Even so, we
support other systems lacking stdint.h, including buildfarm member gaur. Per
a suggestion from Tom Lane. Discussion:
https://postgr.es/m/9598.1550353336@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/04a87ae2626311e922cf964d1e7d42a5ef7d87bf

- Suppress another case of MSVC warning 4146.
https://git.postgresql.org/pg/commitdiff/faee6fae6d09fb5d8c809946a113eb70c2968892

- Fix CLogTruncationLock documentation. Back-patch to v10, which introduced the
lock.
https://git.postgresql.org/pg/commitdiff/301de4f7d92ffe5451d34da2edd36284f3887493

Tatsuo Ishii pushed:

- Doc: remove ancient comment. There's a very old comment in rules.sgml added
back to 2003. It expected to a feature coming back but it never happened. So
now we can safely remove the comment. Back-patched to all supported branches.
Discussion:
https://postgr.es/m/20190211.191004.219630835457494660.t-ishii%40sraoss.co.jp
https://git.postgresql.org/pg/commitdiff/69e5247879291b0164b38d17526658a72884fbe8

Joe Conway pushed:

- Mark pg_config() stable rather than immutable. pg_config() has been marked
immutable since its inception. As part of a larger discussion around the
definition of immutable versus stable and related implications for marking
functions parallel safe raised by Andres, the consensus was clearly that
pg_config() is stable, since it could possibly change output even for the same
minor version with a recompile or installation of a new binary. So mark it
stable. Theoretically this could/should be backpatched, but it was deemed to
be not worth the effort since in practice this is very unlikely to cause
problems in the real world. Discussion:
https://postgr.es/m/20181126234521.rh3grz7aavx2ubjv@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/290e3b77fde9cd063e20d96b1241566a429943de

- Fix documentation for dblink_error_message() return value. The dblink
documentation claims that an empty string is returned if there has been no
error, however OK is actually returned in that case. Also, clarify that an
async error may not be seen unless dblink_is_busy() or dblink_get_result()
have been called first. Backpatch to all supported branches. Reported-by:
realyota Backpatch-through: 9.4 Discussion:
https://postgr.es/m/153371978486.1298.2091761143788088262@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/bc6d7eb689a2d083df981dfd10c65d1a9d32ca64

== Pending Patches ==

Tom Lane sent in a patch to ignore included columns in indxpath.c.

John Naylor sent in another revision of a patch to improve the FSM regression
test and document the fact that functions that use the FSM for will report no
free space for tables without a FSM.

Lætitia Avrot sent in another revision of a patch to add log10 and hyperbolic
functions.

Peter Geoghegan sent in another revision of a patch to make all nbtree entries
unique by having heap TIDs participate in comparisons.

Artur Zakirov sent in a patch to prevent xlogreader from reading a file block
twice.

Konstantin Knizhnik and Dmitry Dolgov traded patches to implement libpq
compression.

Chapman Flack sent in another revision of a patch to fix some infelicities
between SQL/XML and PostgreSQL.

Stephen Frost sent in a patch to integrate GSSAPI encryption with psql.

Takayuki Tsunakawa sent in a patch to speed up transaction completion when any
prior transaction accessed many relations in the same session.

Amit Langote sent in another revision of a patch to speed up planning with
partitions.

Pavel Stěhule sent in another revision of a patch to make LEAST() and GREATEST()
variadic.

Álvaro Herrera sent in two more revisions of a patch to make it possible to
track the progress of CREATE INDEX [CONCURRENTLY].

Haribabu Kommi sent in a patch to pg_basebackup to correct the existing
directory permissions.

Kyotaro HORIGUCHI sent in two more revisions of a patch to protect syscache from
bloating with negative cache entries.

Surafel Temesgen sent in another revision of a patch to add --rows-per-insert to
pg_dump.

Daniel Vérité sent in a patch to remove an unnecessary copy from replace_text().

Michaël Paquier sent in a patch to emit better error messages when lacking
connection slots for autovacuum workers and bgworkers.

Peter Eisentraut sent in a patch to make it possible to slow down WAL-generating
maintenance operations.

Shawn Debnath sent in another revision of a patch to refactor the checkpointer's
fsync request queue.

Andrey Borodin sent in a patch to tweak the LRU cache for shared buffers.

Amit Langote sent in a patch to refactor the grouping_planner.

Michaël Paquier sent in a patch to make sure psql reports extensions created in
temporary schemas.

Amit Langote sent in a patch to remove a no-longer-used argument of
ATAddForeignKeyConstraint.

Masahiko Sawada sent in another revision of a patch to implement block-level
parallel vacuum.

Michael Banck sent in a patch to fix some checksumming-related buglets in the
pg_verify_checksums/pg_basebackup TAP tests.

John Naylor sent in another revision of a patch to reassign high OIDs.

Daisuke Higuchi sent in a patch to fix a problem in ECPG where some CREATE TABLE
... AS ... constructions didn't work.

Evgeniy Efimkin sent in another revision of a patch to add a table filter to
CREATE SUBSCRIPTION.

Etsuro Fujita sent in a patch to fix some costing issues in the PostgreSQL FDW.

Etsuro Fujita sent in a patch to Save PathTargets for distinct/ordered relations
in root->upper_targets[].

Alexey Bashtanov sent in another revision of a patch to make it possible to log
bind parameters on error.

Justin Pryzby sent in a patch to conditionally re-log prepared statement during
execution.

Kyotaro HORIGUCHI sent in another revision of a patch to use shared memory
instead of files for the stats collector.

Amit Langote and Pavel Stěhule traded patches to add support for partitions
(\dP) to psql.

Arseny Sher sent in a patch to pick the latest cmin in
ReorderBufferBuildTupleCidHash.

Michael Banck sent in another revision of a patch to make it possible to verify
page checksums online.

John Naylor sent in another revision of a patch to allow ALTER TABLE on pg_class
and pg_attribute.

Jeff Janes sent in a patch to fix a memory leak introduced by
763f2edd92095b1ca2.

Sergei Kornilov sent in two more revisions of a patch to make it possible to
change walreceiver's conninfo without a restart.

Noah Misch sent in a patch to stop rounding SimpleLruTruncate() cutoffPage
values.

Hugh Ranalli sent in another revision of a patch to generate unaccent rules
remove combining diacritical accents.

Andrey Borodin sent in another revision of a patch to add a GiST verification
function for amcheck.

Fabien COELHO sent in another revision of a patch to fix some libpq
host/hostaddr/conninfo inconsistencies.

Michael Banck sent in another revision of a patch to make it possible to
activate page checksums offline.

Michael Banck sent in another revision of a patch to add a --progress option to
pg_verify_checksums.

Fabien COELHO sent in another revision of a patch to document some gotchas in
pgbench's Zipf distribution.