== PostgreSQL Weekly News - September 24 2017 ==

== PostgreSQL Weekly News - September 24 2017 ==

PostgreSQL 10 Release Candidate 1 released. Test!
https://www.postgresql.org/about/news/1783/

== PostgreSQL Product News ==

pg-wrapper 0.2.2, a wrapper for PHP's pgsql extension, released.
https://github.com/sad-spirit/pg-wrapper

brimbox 2.3.4, a web database interface and framework built atop PostgreSQL, released.
https://www.brimbox.com/brimbox-version-2-3-4-released/

pg_dirtyread 1.2, an extension that provides the ability to read dead but
unvacuumed rows from a relation, released.
https://github.com/ChristophBerg/pg_dirtyread

pg_builder 0.2.1, a PHP query builder for PostgreSQL, released.
https://github.com/sad-spirit/pg-builder

== PostgreSQL Jobs for September ==

http://archives.postgresql.org/pgsql-jobs/2017-09/

== PostgreSQL Local ==

PGDay.IT 2017 will take place October 13th, in Milan, Italy.
http://pgday.it

PostgreSQL Conference Europe 2017 will be held on October 24-27, 2017 in the
Warsaw Marriott Hotel, in Warsaw, Poland.
https://2017.pgconf.eu/

PGConf Local: Seattle will be held November 13 - 14, 2017.
https://www.pgconf.us/#Seattle2017

PGDay Australia 2017 will be held on November 17 in Melbourne.
http://2017.pgday.com.au/

2ndQuadrant PostgreSQL Conference 2017 (2Q PGConf, for short) will be hosted on
November 6th & 7th in New York City, and November 9th in Chicago.
http://www.2qpgconf.com/

PostgreSQL Session November 17th, 2017, in Paris, France. The CfP is open until
September 30, 2017 at call-for-paper AT postgresql-sessions DOT org. Details at
http://blog.dalibo.com/2017/08/29/cfp_pgsession9.html

PGConf Local: Austin will be held December 4 - 5, 2017. Call for Papers is
now open at https://www.pgconf.us/conferences/Austin2017

PGConf.ASIA 2017 will take place on December 4-6 2017 in Akihabara, Tokyo,
Japan.
http://www.pgconf.asia/EN/2017/

PGConf India 2018 will be on February 22-23, 2018 in Bengaluru, Karnataka.
Proposals are due via https://goo.gl/forms/F9hRjOIsaNasVOAz2 by October 31st, 2017.
http://pgconf.in/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm EST5EDT. Please send English
language ones to david(at)fetter(dot)org, German language to pwn(at)pgug(dot)de, Italian
language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Peter Eisentraut pushed:

- Remove dead external links from documentation.
https://git.postgresql.org/pg/commitdiff/d31892e2105cf48d8430807d74d5fdf1434af541

- Update some dead external links in the documentation.
https://git.postgresql.org/pg/commitdiff/4b17c894293d0c3ed944da76aeb9bc2bb02a6db6

- Sync process names between ps and pg_stat_activity. Remove gratuitous
differences in the process names shown in pg_stat_activity.backend_type and
the ps output. Reviewed-by: Takayuki Tsunakawa
<tsunakawa(dot)takay(at)jp(dot)fujitsu(dot)com>
https://git.postgresql.org/pg/commitdiff/be87b70b6117609b7db0cd5e0cd96e7f569bdddb

- Fix compiler warning. from gcc-7 -Wformat-truncation (via -Wall)
https://git.postgresql.org/pg/commitdiff/d42294fc00da4b97d04ddb4401b76295e8d86816

- Add basic TAP test setup for pg_upgrade. The plan is to convert the current
pg_upgrade test to the TAP framework. This commit just puts a basic TAP test
in place so that we can see how the build farm behaves, since the build farm
client has some special knowledge of the pg_upgrade tests. Author: Michael
Paquier <michael(dot)paquier(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/f41e56c76e39f02bef7ba002c9de03d62b76de4d

- Fix build with !USE_WIDE_UPPER_LOWER. The placement of the ifdef blocks in
formatting.c was pretty bogus, so the code failed to compile if
USE_WIDE_UPPER_LOWER was not defined. Reported-by: Peter Geoghegan
<pg(at)bowt(dot)ie> Reported-by: Noah Misch <noah(at)leadboat(dot)com>
https://git.postgresql.org/pg/commitdiff/e6023ee7fa73a2d9a2d7524f63584844b2291def

- Remove contrib/chkpass. The recent addition of a test suite for this module
revealed a few problems. It uses a crypt() method that is no longer
considered secure and doesn't work anymore on some platforms. Using a
volatile input function violates internal sanity check assumptions and leads
to failures on the build farm. So this module is neither a usable security
tool nor a good example for an extension. No one wanted to argue for keeping
or improving it, so remove it. Discussion:
https://www.postgresql.org/message-id/5645b0d7-cc40-6ab5-c553-292a91091ee7%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/5d3cad564729f64d972c5c803ff34f0eb40bfd0c

- doc: Document commands that cannot be run in a transaction block. Mainly
covering the new CREATE SUBSCRIPTION and DROP SUBSCRIPTION, but ALTER DATABASE
SET TABLESPACE was also missing.
https://git.postgresql.org/pg/commitdiff/91ad8b416cee753eaa6f520ee2d21c2d41853381

- Revert "Add basic TAP test setup for pg_upgrade". This reverts commit
f41e56c76e39f02bef7ba002c9de03d62b76de4d. The build farm client would run the
pg_upgrade tests twice, once as part of the existing pg_upgrade check run and
once as part of picking up all TAP tests by looking for "t" directories.
Since the pg_upgrade tests are pretty slow, we will need a better solution or
possibly a build farm client change before we can proceed with this.
https://git.postgresql.org/pg/commitdiff/58ffe141eb37c3f027acd25c1fc6b36513bf9380

- Fix saving and restoring umask. In two cases, we set a different umask for
some piece of code and restore it afterwards. But if the contained code
errors out, the umask is not restored. So add TRY/CATCH blocks to fix that.
https://git.postgresql.org/pg/commitdiff/aa6b7b72d9bcf967cbccd378de4bc5cef33d02f9

- Refactor new file permission handling. The file handling functions from fd.c
were called with a diverse mix of notations for the file permissions when they
were opening new files. Almost all files created by the server should have
the same permissions set. So change the API so that e.g. OpenTransientFile()
automatically uses the standard permissions set, and OpenTransientFilePerm()
is a new function that takes an explicit permissions set for the few cases
where it is needed. This also saves an unnecessary argument for call sites
that are just opening an existing file. While we're reviewing these APIs, get
rid of the FileName typedef and use the standard const char * for the file
name and mode_t for the file mode. This makes these functions match other
file handling functions and removes an unnecessary layer of mysteriousness.
We can also get rid of a few casts that way. Author: David Steele
<david(at)pgmasters(dot)net>
https://git.postgresql.org/pg/commitdiff/0c5803b450e0cc29b3527df3f352e6f18a038cc6

- Fix pg_basebackup test to original intent. One test case was meant to check
that pg_basebackup does not succeed when a slot is specified with -S but WAL
streaming is not selected, which used to require specifying -X stream. Since
-X stream is the default in PostgreSQL 10, this test case no longer covers
that meaning, but the pg_basebackup invocation happened to fail anyway for the
unrelated reason that the specified replication slot does not exist. To fix,
move the test case to later in the file where the slot does exist, and add -X
none to the invocation so that it covers the originally meant behavior.
extracted from a patch by Michael Banck <michael(dot)banck(at)credativ(dot)de>
https://git.postgresql.org/pg/commitdiff/74ca8f9b9077017529fe658e445a11da296ac6ab

- doc: Expand user documentation on SCRAM. Explain more about how the different
password authentication methods and the password_encryption settings relate to
each other, give some upgrading advice, and set a better link from the release
notes. Reviewed-by: Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/9b31c72a9492880e657b68b1ed971dec3c361c95

- Allow ICU to use SortSupport on Windows with UTF-8. There is no reason to
ever prevent the use of SortSupport on Windows when ICU locales are used. We
previously avoided SortSupport on Windows with UTF-8 server encoding and a non
C-locale due to restrictions in Windows' libc functionality. This is now
considered to be a restriction in one platform's libc collation provider, and
not a more general platform restriction. Reported-by: Peter Geoghegan
<pg(at)bowt(dot)ie>
https://git.postgresql.org/pg/commitdiff/6dda0998afc7d449145b9ba216844bdba7a817d6

Tom Lane pushed:

- Fix, or at least ameliorate, bugs in logicalrep_worker_launch(). If we failed
to get a background worker slot, the code just walked away from the
logicalrep-worker slot it already had, leaving that looking like the worker is
still starting up. This led to an indefinite hang in subscription startup, as
reported by Thomas Munro. We must release the slot on failure. Also fix a
thinko: we must capture the worker slot's generation before releasing
LogicalRepWorkerLock the first time, else testing to see if it's changed is
pretty meaningless. BTW, the CHECK_FOR_INTERRUPTS() in
WaitForReplicationWorkerAttach is a ticking time bomb, even without
considering the possibility of elog(ERROR) in one of the other functions it
calls. Really, this entire business needs a redesign with some actual thought
about error recovery. But for now I'm just band-aiding the case observed in
testing. Back-patch to v10 where this code was added. Discussion:
https://postgr.es/m/CAEepm=2bP3TBMFBArP6o20AZaRduWjMnjCjt22hSdnA-EvrtCw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3e1683d37e1d751eb2df9a5cb0507bebc6cf7d05

- Make DatumGetFoo/PG_GETARG_FOO/PG_RETURN_FOO macro names more consistent. By
project convention, these names should include "P" when dealing with a pointer
type; that is, if the result of a GETARG macro is of type FOO *, it should be
called PG_GETARG_FOO_P not just PG_GETARG_FOO. Some newer types such as JSONB
and ranges had not followed the convention, and a number of contrib modules
hadn't gotten that memo either. Rename the offending macros to improve
consistency. In passing, fix a few places that thought PG_DETOAST_DATUM()
returns a Datum; it does not, it returns "struct varlena *". Applying
DatumGetPointer to that happens not to cause any bad effects today, but it's
formally wrong. Also, adjust an ltree macro that was designed without any
thought for what pgindent would do with it. This is all cosmetic and
shouldn't have any impact on generated code. Mark Dilger, some further tweaks
by me Discussion:
https://postgr.es/m/EA5676F4-766F-4F38-8348-ECC7DB427C6A@gmail.com
https://git.postgresql.org/pg/commitdiff/4bd1994650fddf49e717e35f1930d62208845974

- Make ExplainOpenGroup and ExplainCloseGroup public. Extensions with custom
plan nodes might like to use these in their EXPLAIN output. Hadi Moshayedi
Discussion:
https://postgr.es/m/CA+_kT_dU-rHCN0u6pjA6bN5CZniMfD=-wVqPY4QLrKUY_uJq5w@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/66917bfaa7bb0b6bae52a5fe631a8b6443203f55

- Minor code-cleanliness improvements for btree. Make the btree page-flags test
macros (P_ISLEAF and friends) return clean boolean values, rather than values
that might not fit in a bool. Use them in a few places that were randomly
referencing the flag bits directly. In passing, change access/nbtree/'s only
direct use of BUFFER_LOCK_SHARE to BT_READ. (Some think we should go the
other way, but as long as we have BT_READ/BT_WRITE, let's use them
consistently.) Masahiko Sawada, reviewed by Doug Doole Discussion:
https://postgr.es/m/CAD21AoBmWPeN=WBB5Jvyz_Nt3rmW1ebUyAnk3ZbJP3RMXALJog@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/eb5c404b17752ca566947f12cb702438dcccdcb1

- Cache datatype-output-function lookup info across calls of concat(). Testing
indicates this can save a third to a half of the runtime of the function.
Pavel Stehule, reviewed by Alexander Kuzmenkov Discussion:
https://postgr.es/m/CAFj8pRAT62pRgjoHbgTfJUc2uLmeQ4saUj+yVJAEZUiMwNCmdg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ed22fb8b0091deea23747310fa7609079a96cf82

- Remove no-op GiST support functions in the core GiST opclasses. The preceding
patch allowed us to remove useless GiST support functions. This patch
actually does that for all the no-op cases in the core GiST code. This buys
us whatever performance gain is to be had, and more importantly exercises the
preceding patch. There remain no-op functions in the contrib GiST opclasses,
but those will take more work to remove. Discussion:
https://postgr.es/m/CAJEAwVELVx9gYscpE=Be6iJxvdW5unZ_LkcAaVNSeOwvdwtD=A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2d484f9b058035d41204f2eb8a0a8d2e8ee57b44

- Allow no-op GiST support functions to be omitted. There are common use-cases
in which the compress and/or decompress functions can be omitted, with the
result being that we make no data transformation when storing or retrieving
index values. Previously, you had to provide a no-op function anyway, but
this patch allows such opclass support functions to be omitted. Furthermore,
if the compress function is omitted, then the core code knows that the stored
representation is the same as the original data. This means we can allow
index-only scans without requiring a fetch function to be provided either.
Previously you had to provide a no-op fetch function if you wanted IOS to
work. This reportedly provides a small performance benefit in such cases, but
IMO the real reason for doing it is just to reduce the amount of useless
boilerplate code that has to be written for GiST opclasses. Andrey Borodin,
reviewed by Dmitriy Sarafannikov Discussion:
https://postgr.es/m/CAJEAwVELVx9gYscpE=Be6iJxvdW5unZ_LkcAaVNSeOwvdwtD=A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d3a4f89d8a3e500bd7c0b7a8a8a5ce1b47859128

- Fix erroneous documentation about noise word GROUP. GRANT, REVOKE, and some
allied commands allow the noise word GROUP before a role name (cf. grantee
production in gram.y). This option does not exist elsewhere, but it had
nonetheless snuck into the documentation for ALTER ROLE, ALTER USER, and
CREATE SCHEMA. Seems to be a copy-and-pasteo in commit 31eae6028, which did
expand the syntax choices here, but not in that way. Back-patch to 9.5 where
that came in. Discussion:
https://postgr.es/m/20170916123750.8885.66941@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/36b564c648a044e42ca461466ae14d8588e6c5e2

- Fix instability in subscription regression test. 005_encoding.pl neglected to
wait for the subscriber's initial synchronization to happen. While we have
not seen this fail in the buildfarm, it's pretty easy to demonstrate there's
an issue by hacking logicalrep_worker_launch() to fail most of the time.
Michael Paquier Discussion: https://postgr.es/m/27032.1505749806@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/4939488af9b86edfff9b981773cd388d361c5830

- Improve dubious memory management in pg_newlocale_from_collation().
pg_newlocale_from_collation() used malloc() and strdup() directly, which is
generally not per backend coding style, and it didn't bother to check for
failure results, but would just SIGSEGV instead. Also, if one of the numerous
error checks in the middle of the function failed, the already-allocated
memory would be leaked permanently. Admittedly, it's not a lot of memory, but
it could build up if this function were called repeatedly for a bad collation.
The first two problems are easily cured by palloc'ing in TopMemoryContext
instead of calling libc directly. We can fairly easily dodge the leakage
problem for the struct pg_locale_struct by filling in a temporary variable and
allocating permanent storage only once we reach the bottom of the function.
It's harder to get rid of the potential leakage for ICU's copy of the
collcollate string, but at least that's only allocated after most of the error
checks; so live with that aspect. Back-patch to v10 where this code came in,
with one or another of the ICU patches.
https://git.postgresql.org/pg/commitdiff/7b86c2ac9563ffd9b870cfd73a769431b7922e81

- Give a better error for duplicate entries in VACUUM/ANALYZE column list.
Previously, the code didn't think about this case and would just try to
analyze such a column twice. That would fail at the point of inserting the
second version of the pg_statistic row, with obscure error messsages like
"duplicate key value violates unique constraint" or "tuple already updated by
self", depending on context and PG version. We could allow the case by
ignoring duplicate column specifications, but it seems better to reject it
explicitly. The bogus error messages seem like arguably a bug, so back-patch
to all supported versions. Nathan Bossart, per a report from Michael Paquier,
and whacked around a bit by me. Discussion:
https://postgr.es/m/E061A8E3-5E3D-494D-94F0-E8A9B312BBFC@amazon.com
https://git.postgresql.org/pg/commitdiff/71480501057fee9fa3649b072173ff10e2b842d0

- Revert "Fix bool/int type confusion". This reverts commit
0ec2e908babfbfde83a3925680f06b16408739ff. We'll use the upstream (IANA) fix
instead.
https://git.postgresql.org/pg/commitdiff/a890432a872afc9ca2327573f3313fd994d17384

- Sync our copy of the timezone library with IANA tzcode master. This patch
absorbs a few unreleased fixes in the IANA code. It corresponds to commit
2d8b944c1cec0808ac4f7a9ee1a463c28f9cd00a in https://github.com/eggert/tz.
Non-cosmetic changes include: TZDEFRULESTRING is updated to match current US
DST practice, rather than what it was over ten years ago. This only matters
for interpretation of POSIX-style zone names (e.g., "EST5EDT"), and only if
the timezone database doesn't include either an exact match for the zone name
or a "posixrules" entry. The latter should not be true in any current
Postgres installation, but this could possibly matter when using
--with-system-tzdata. Get rid of a nonportable use of "++var" on a bool var.
This is part of a larger fix that eliminates some vestigial support for
consecutive leap seconds, and adds checks to the "zic" compiler that the data
files do not specify that. Remove a couple of ancient compatibility hacks.
The IANA crew think these are obsolete, and I tend to agree. But perhaps our
buildfarm will think different. Back-patch to all supported branches, in line
with our policy that all branches should be using current IANA code. Before
v10, this includes application of current pgindent rules, to avoid whitespace
problems in future back-patches. Discussion:
https://postgr.es/m/E1dsWhf-0000pT-F9@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/47f849a3c9005852926dca551d70ad8111f09f3a

- Assume wcstombs(), towlower(), and sibling functions are always present.
These functions are required by SUS v2, which is our minimum baseline for Unix
platforms, and are present on all interesting Windows versions as well. Even
our oldest buildfarm members have them. Thus, we were not testing the
"!USE_WIDE_UPPER_LOWER" code paths, which explains why the bug fixed in commit
e6023ee7f escaped detection. Per discussion, there seems to be no more
real-world value in maintaining this option. Hence, remove the configure-time
tests for wcstombs() and towlower(), remove the USE_WIDE_UPPER_LOWER symbol,
and remove all the !USE_WIDE_UPPER_LOWER code. There's not actually all that
much of the latter, but simplifying the #if nests is a win in itself.
Discussion: https://postgr.es/m/20170921052928.GA188913@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/85feb77aa09cda9ff3e12cf95c757c499dc25343

- Mop-up for commit 85feb77aa09cda9ff3e12cf95c757c499dc25343. Adjust commentary
in regc_pg_locale.c to remove mention of the possibility of not having
<wctype.h> functions, since we no longer consider that. Eliminate duplicate
code in wparser_def.c by generalizing the p_iswhat macro to take a parameter
saying what to return for non-ASCII chars in C locale. (That's not really a
consequence of the USE_WIDE_UPPER_LOWER-ectomy, but I noticed it while doing
that.)
https://git.postgresql.org/pg/commitdiff/ed87e1980706975e7aa412bee200087774c5ff22

- Allow up to 3 "-P 1" reports per thread in pgbench run of 2 seconds. There
seems to be some considerable imprecision in the timing of -P progress
reports. Nominally each thread ought to produce 2 reports in this test, but
about 10% of the time we only get one, and 1% of the time we get three, as per
buildfarm results so far. Pending further investigation, treat the last case
as a "pass". (I, tgl, am suspicious that this still might not be lax enough,
now that it's obvious that the behavior is load-dependent; but there's not yet
buildfarm evidence to confirm that suspicion.) Fabien Coelho Discussion:
https://postgr.es/m/26654.1505232433@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/0f574a7afb5c998d19dc3d981e45cb10267286ed

- Ten-second timeout in 013_crash_restart.pl is not enough, let's try 60. Per
buildfarm member topminnow.
https://git.postgresql.org/pg/commitdiff/01c7d3ef85d4b0e1c52cc1a3542864f95f386f76

- Improve memory management in autovacuum.c. Invoke vacuum(), as well as "work
item" processing, in the PortalContext that do_autovacuum() has manufactured,
which will be reset before each such invocation. This ensures cleanup of any
memory leaked by these operations. It also avoids the rather dangerous
practice of calling vacuum() in a context that vacuum() itself will destroy
while it runs. There's no known live bug there, but it's not hard to imagine
introducing one if we leave it like this. Tom Lane, reviewed by Michael
Paquier and Alvaro Herrera Discussion:
https://postgr.es/m/13849.1506114543@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/335f3d04e4c8dd495c4dd30ab1049b6fe8f25052

- Fix bogus size calculation in strlist_to_textarray(). It's making an array of
Datum, not an array of text *. The mistake is harmless since those are
currently the same size, but it's still wrong.
https://git.postgresql.org/pg/commitdiff/737639017c87d5a0a466e8676f1eadc61d775c78

- ... and the very same bug in publicationListToArray(). Sigh.
https://git.postgresql.org/pg/commitdiff/24541ffd788d56009126fff52b2341ada6c84245

- Remove pgbench "progress" test pending solution of its timing issues.
Buildfarm member skink shows that this is even more flaky than I thought.
There are probably some actual pgbench bugs here as well as a timing
dependency. But we can't have stuff this unstable in the buildfarm, it
obscures other issues.
https://git.postgresql.org/pg/commitdiff/ad51c6fb5708342e603d12a730bbc4e663bd637e

- Fix assorted infelicities in new SetWALSegSize() function. * Failure to check
for malloc failure (ok, pretty unlikely here, but that's not an excuse). *
Leakage of open fd on read error, and of malloc'd buffer always. * Incorrect
assumption that a short read would set errno to zero. * Failure to adhere to
message style conventions (in particular, not reporting errno where relevant;
using "couldn't open" rather than "could not open" is not really in line with
project style either). * Missing newlines on some messages. Coverity spotted
the leak problems; I noticed the rest while fixing the leaks.
https://git.postgresql.org/pg/commitdiff/8485a25a8c9a419ff3e0d30e43e4abd5e680cc65

Andres Freund pushed:

- Fix crash restart bug introduced in 8356753c212. The bug was caused by not
re-reading the control file during crash recovery restarts, which lead to an
attempt to pfree() shared memory contents. The fix is to re-read the control
file, which seems good anyway. It's unclear as of this moment, whether we
want to keep the refactoring introduced in the commit referenced above, or
come up with an alternative approach. But fixing the bug in the mean time
seems like a good idea regardless. A followup commit will introduce
regression test coverage for crash restarts. Reported-By: Tom Lane
Discussion: https://postgr.es/m/14134.1505572349@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/ec9e05b3c392ba9587f283507459737684539574

- Fix uninitialized variable in dshash.c. A bugfix for commit
8c0d7bafad36434cb08ac2c78e69ae72c194ca20. The code would have crashed if
hashtable->size_log2 ever had the same value as hashtable->control->size_log2
by coincidence. Per Valgrind. Author: Thomas Munro Reported-By: Tomas Vondra
Discussion:
https://postgr.es/m/e72fb33c-4f31-f276-e972-263d9b59554d%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/0fb9e4ace5ce4d479d839a720f32b99fdc87f455

- Rearm statement_timeout after each executed query. Previously
statement_timeout, in the extended protocol, affected all messages till a Sync
message. For clients that pipeline/batch query execution that's problematic.
Instead disable timeout after each Execute message, and enable, if necessary,
the timer in start_xact_command(). As that's done only for Execute and not
Parse / Bind, pipelining the latter two could still cause undesirable
timeouts. But a survey of protocol implementations shows that all drivers
issue Sync messages when preparing, and adding timeout rearming to both is
fairly expensive for the common parse / bind / execute sequence. Author:
Tatsuo Ishii, editorialized by Andres Freund Reviewed-By: Takayuki Tsunakawa,
Andres Freund Discussion:
https://postgr.es/m/20170222.115044.1665674502985097185.t-ishii@sraoss.co.jp
https://git.postgresql.org/pg/commitdiff/f8e5f156b30efee5d0038b03e38735773abcb7ed

- Add test for postmaster crash restarts. Given that I managed to break this...
We probably should extend the tests to also cover other sub-processes dying,
but that's something for later. Author: Andres Freund Discussion:
https://postgr.es/m/20170917080752.rcmihzfmgbeuqjk2@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/a1924a4ea29399111e5155532ca24c9c51d3c82d

- Speedup pgstat_report_activity by moving mb-aware truncation to read side.
Previously multi-byte aware truncation was done on every
pgstat_report_activity() call - proving to be a bottleneck for workloads with
long query strings that execute quickly. Instead move the truncation to the
read side, which commonly is executed far less frequently. That's possible
because all server encodings allow to determine the length of a multi-byte
string from the first byte. Rename PgBackendStatus.st_activity to
st_activity_raw so existing extension users of the field break - their code
has to be adjusted to use pgstat_clip_activity(). Author: Andres Freund
Tested-By: Khuntal Ghosh Reviewed-By: Robert Haas, Tom Lane Discussion:
https://postgr.es/m/20170912071948.pa7igbpkkkviecpz@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/54b6cd589ac2f5635a42511236a5eb7299e2dcaf

- Avoid use of non-portable strnlen() in pgstat_clip_activity(). The use of
strnlen rather than strlen was just paranoia. Instead of giving up on the
paranoia, just implement the safeguard differently. And add a comment
explaining why we're careful. Author: Andres Freund Discussion:
https://postgr.es/m/E1duOkJ-0001Mc-U5@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/71edbb6f66f7139d6209334ef8734a122ba06b56

- s/NULL byte/NUL byte/ in comment refering to C string terminator.
Reported-By: Robert Haas Discussion:
https://postgr.es/m/CA+Tgmoa+YBvWgFST2NVoeXjVSohEpK=vqnVCsoCkhTVVxfLcVQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/896537f078ba4d709ce754ecaff8350fd55bdfd8

- Make WAL segment size configurable at initdb time. For performance reasons a
larger segment size than the default 16MB can be useful. A larger segment size
has two main benefits: Firstly, in setups using archiving, it makes it easier
to write scripts that can keep up with higher amounts of WAL, secondly, the
WAL has to be written and synced to disk less frequently. But at the same
time large segment size are disadvantageous for smaller databases. So far the
segment size had to be configured at compile time, often making it unrealistic
to choose one fitting to a particularly load. Therefore change it to a initdb
time setting. This includes a breaking changes to the xlogreader.h API, which
now requires the current segment size to be configured. For that and similar
reasons a number of binaries had to be taught how to recognize the current
segment size. Author: Beena Emerson, editorialized by Andres Freund
Reviewed-By: Andres Freund, David Steele, Kuntal Ghosh, Michael Paquier, Peter
Eisentraut, Robert Hass, Tushar Ahuja Discussion:
https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/fc49e24fa69a15efacd5b8958115ed9c43c48f9a

- Make new crash restart test a bit more robust. Add timeouts in case psql
doesn't deliver the expected output, and try to cause the monitoring psql to
be fully connected to a backend. This isn't necessarily everything needed,
but at least the timeouts should reduce the pain for buildfarm owners.
Author: Andres Freund Reported-By: Tom Lane, BF animals prairiedog and
calliphoridae Discussion:
https://postgr.es/m/E1du6ZT-00043I-91@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/1910353675bd149e1020b29c0fae02538fc358cd

- Accept that server might not be able to send error in crash recovery test. As
it turns out we can't rely that the script's monitoring session is terminated
with a proper error by the server, because the session might be terminated
while already trying to send data. Also improve robustness and error
reporting facilities of the test, developed while debugging this issue.
Discussion:
https://postgr.es/m/20170920020038.kllxgilo7xzwmtto@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/5ada1fcd0c30be1b0b793a802cf6da386a6c1925

- Fix s/intidb/initdb/ typo. Reported-By: Michael Paquier Discussion:
https://postgr.es/m/CAB7nPqTfaKAYZ4wuUM-W8kc4VnXrxX1=5-a9i==VoUPTMFpsgg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/f9583e86b4bfa8c4e4d83ab33e5dcdaeab5c45a1

- Expand expected output for recovery test even further. I'd assumed that the
backend being killed should be able to get out an error message - but it turns
out it's not guaranteed that it's not still sending a ready-for-query. Really
need to do something about getting these error message to the client.
Reported-By: Thomas Munro, Tom Lane Discussion:
https://postgr.es/m/CAEepm=0TE90nded+bNthP45_PEvGAAr=3gxhHJObL4xmOLtX0w@mail.gmail.com
https://postgr.es/m/14968.1506101414@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/8d926029e817d280b2376433e3aaa3895e1a7128

- Add inline murmurhash32(uint32) function. The function already existed in
tidbitmap.c but more users requiring fast hashing of 32bit ints are coming up.
Author: Andres Freund Discussion:
https://postgr.es/m/20170914061207.zxotvyopetm7lrrp@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/791961f59b792fbd4f0a992d3ccab47298e79103

Andrew Dunstan pushed:

- Add citext_pattern_ops for citext contrib module. This is similar to
text_pattern_ops. Alexey Chernyshov, reviewed by Jacob Champion.
https://git.postgresql.org/pg/commitdiff/f2464997644c64b5dec93ab3c08305f48bfe14f1

- Set client encoding to UTF8 for the citext regression script. Problem
introduced with non-ascii characters in commit f2464997644c and discovered on
various buildfarm animals.
https://git.postgresql.org/pg/commitdiff/890faaf1957759c6e17fbcbfd16f7cabc4a59d07

- Disable multi-byte citext tests. This reverts commit 890faaf1 which attempted
unsuccessfully to deal with the problem, and instead just comments out these
tests like other similar tests elsewhere in the script.
https://git.postgresql.org/pg/commitdiff/d1687c6926819f023c78b353458950a303796aba

- Quieten warnings about unused variables. These variables are only ever
written to in assertion-enabled builds, and the latest Microsoft compilers
complain about such variables in non-assertion-enabled builds. Apparently
they don't worry so much about variables that are written to but not read
from, so most of our PG_USED_FOR_ASSERTS_ONLY variables don't cause the
problem. Discussion: https://postgr.es/m/7800.1505950322@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/28ae524bbf865d23eb10f6ae1b996d59dcc30e4e

- Provide a test for variable existence in psql. "\if :{?variable_name}" will
be translated to "\if TRUE" if the variable exists and "\if FALSE" otherwise.
Thus it will be possible to execute code conditionally on the existence of the
variable, regardless of its value. Fabien Coelho, with some review by Robins
Tharakan and some light text editing by me. Discussion:
https://postgr.es/m/alpine.DEB.2.20.1708260835520.3627@lancre
https://git.postgresql.org/pg/commitdiff/d57c7a7c506276597af619bdb8c62fa5b592745a

Bruce Momjian pushed:

- doc: add example of % substitution for connection URIs. Reported-by: Zhou
Digoal Discussion:
https://postgr.es/m/20170912133722.25637.91@wrigleys.postgresql.org
Backpatch-through: 10
https://git.postgresql.org/pg/commitdiff/d61f5bb7c444255b064a60df782907f7dddad61a

- docs: re-add instructions on setting wal_level for rsync use. This step was
erroneously removed four days ago by me. Reported-by: Magnus via IM
Backpatch-through: 9.5
https://git.postgresql.org/pg/commitdiff/00210e3fb974ff2b9affc4d8f3b29f9cb3645a60

Magnus Hagander pushed:

- Mention need for --no-inc-recursive in rsync command. Since rsync 3.0.0
(released in 2008), the default way to enumerate changes was changed in a way
that makes it less likely that the hardlink sync mode works. Since the whole
point of the documented procedure is for the hardlinks to work, change our
docs to suggest using the backwards compatibility switch.
https://git.postgresql.org/pg/commitdiff/2c74e6c1dcc5002fa8b822e5757f6c95d899fb7a

Robert Haas pushed:

- Fix typo. Thomas Munro Discussion:
http://postgr.es/m/CAEepm=2j-HAgnBUrAazwS0ry7Z_ihk+d7g+Ye3u99+6WbiGt_Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7f3a3312abf34ea7e899046e326775612802764b

- Fix create_lateral_join_info to handle dead relations properly. Commit
0a480502b092195a9b25a2f0f199a21d592a9c57 broke it. Report by Andreas
Seltenreich. Fix by Ashutosh Bapat. Discussion:
http://postgr.es/m/874ls2vrnx.fsf@ansel.ydns.eu
https://git.postgresql.org/pg/commitdiff/57eebca03a9eb61eb18f8ea9db94775653f797d1

- Associate partitioning information with each RelOptInfo. This is not used for
anything yet, but it is necessary infrastructure for partition-wise join and
for partition pruning without constraint exclusion. Ashutosh Bapat, reviewed
by Amit Langote and with quite a few changes, mostly cosmetic, by me.
Additional review and testing of this patch series by Antonin Houska, Amit
Khandekar, Rafia Sabih, Rajkumar Raghuwanshi, Thomas Munro, and Dilip Kumar.
Discussion:
http://postgr.es/m/CAFjFpRfneFG3H+F6BaiXemMrKF+FY-POpx3Ocy+RiH3yBmXSNw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/9140cf8269b0c4ae002b2748d93979d535891311

- hash: Implement page-at-a-time scan. Commit
09cb5c0e7d6fbc9dee26dc429e4fc0f2a88e5272 added a similar optimization to btree
back in 2006, but nobody bothered to implement the same thing for hash
indexes, probably because they weren't WAL-logged and had lots of other
performance problems as well. As with the corresponding btree case, this
eliminates the problem of potentially needing to refind our position within
the page, and cuts down on pin/unpin traffic as well. Ashutosh Sharma,
reviewed by Alexander Korotkov, Jesper Pedersen, Amit Kapila, and me. Some
final edits to comments and README by me. Discussion:
http://postgr.es/m/CAE9k0Pm3KTx93K8_5j6VMzG4h5F+SyknxUwXrN-zqSZ9X8ZS3w@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7c75ef571579a3ad7a1d3ee909f11dba5e0b9440

- For wal_consistency_checking, mask page checksum as well as page LSN. If the
LSN is different, the checksum will be different, too. Ashwin Agrawal,
reviewed by Michael Paquier and Kuntal Ghosh Discussion:
http://postgr.es/m/CALfoeis5iqrAU-+JAN+ZzXkpPr7+-0OAGv7QUHwFn=-wDy4o4Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/6a2fa09c0cba0e5a11854d733872ac18511f4c83

Álvaro Herrera pushed:

- Document further existing locks as wait events. Reported-by: Jeremy Schneider
Author: Michael Paquier Discussion:
https://postgr.es/m/CA+fnDAZaPCwfY8Lp-pfLnUGFAXRu1VfLyRgdup-L-kwcBj8MqQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/885cab58115a5af9484926ddee8dca3dc0106c1e

- Test BRIN autosummarization. There was no coverage for this code.
Reported-by: Nikolay Shaplov, Tom Lane Discussion:
https://postgr.es/m/2700647.XEouBYNZic@x200m
https://postgr.es/m/13849.1506114543@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/404ba54e8fd3036eee0f9241f68b17092ce734ee

== Pending Patches ==

Rushabh Lathia sent in another revision of a patch to fix an error that
manifested as GatherMerge failing to push a target list down.

Jesper Pedersen sent in another revision of a patch to implement hash
partitioning.

Oliver Ford sent in another revision of a patch to add Roman numeral conversion
to to_number.

Dilip Kumar sent in another revision of a patch to prune partitions more
efficiently in queries.

Ashutosh Bapat sent in a patch to skip "dead" relations in
create_lateral_join_info().

Jeevan Chalke sent in another revision of a patch to implement partition-wise
aggregation/grouping for declaratively partitioned tables.

Kyotaro HORIGUCHI sent in another revision of a patch to fix a race condition
between SELECT and ALTER TABLE NO INHERIT.

Noah Misch sent in a patch to change the release notes to do much less invasive
surgery on the information_schema.table_privileges view, replacing it rather
than re-creating the information schema whole cloth.

Michaël Paquier sent in another revision of a patch to remove
ALLOW_DANGEROUS_LO_FUNCTIONS for LO-related superuser checks, replace superuser
checks of large object import/export by ACL checks, remove ACL checks for large
objects when opening them.

Haribabu Kommi sent in another revision of a patch to add the support of storage
handler as an access method, add storage AM API hooks and related functions, add
storageam handler to the relation structure, add a tuple visibility function,
slot hooks, a tuple insert API, and scan functions to the storage AM, and remove
HeapScanDesc uses outside heap-specific code.

Rushabh Lathia sent in another revision of a patch to add parallel B-tree index
build sorting.

Jeff Janes sent in another revision of a patch to ensure that parallel paths
include target list cost.

Thomas Munro sent in another revision of a patch to make SERIALIZABLE isolation
work with parallel operations.

Masahiko Sawada sent in three more revisions of a patch to enable custom
initialization for pgbench.

Ashutosh Bapat sent in two more revisions of a patch to implement partition-wise
join for joins between (declaratively) partitioned tables.

Michaël Paquier sent in another revision of a patch to rewrite tests of
pg_upgrade as TAP tests.

Craig Ringer sent in a patch to add a utility class for value and identifier
quoting for use in TAP tests.

Vaishnavi Prabakaran sent in a patch to correct the debug log to reflect
the change of function name from lo_open to be_lo_open.

Amit Khandekar sent in another revision of a patch to implement parallel append.

Andres Freund sent in a patch to add the ability to force libpq to negotiate a
specific version of the protocol and add minimal v2 protocol regression tests
based on same.

Alexander Korotkov sent in two more revisions of a patch to add SP-GiST compress
methods, and GiST support for circles and polygons.

Masahiko Sawada sent in three revisions of a patch to fix an assertion failure
when the non-exclusive pg_stop_backup aborted.

Amul Sul sent in a patch to only handle oid sysattr, to reduce overhead in a
patch set intended to improve syscache/catcache performance.

Rajkumar Raghuwanshi sent in a patch to add some tests for partition-wise join
between declaratively partitioned tables.

Thomas Munro sent in a shell program which shows which blocks of SQL hit which
lines of C.

Peter Eisentraut sent in another revision of a patch to run only top-level
recursive lcov, have lcov exclude external files, add lcov --initial, add the
PostgreSQL version to coverage output, remove the coverage details view, run
coverage commands quietly, improve vpath support in the plperl build, support
coverage on vpath builds, and exclude flex-generated code from coverage testing.

Ashutosh Sharma sent in three more revisions of a patch to rewrite hash index
scan to work page at a time, remove redundant hash function _hash_step and do
some code cleanup, and improve the locking startegy during VACUUM in hash index
for regular tables.

Lucas Fittl sent in a patch to only skip query cancel itself when query cancel
holdoff count is positive.

Rafia Sabih sent in another revision of a patch to make gather faster.

Tom Lane sent in another revision of a patch to support arrays of domains.

Kyotaro HORIGUCHI sent in a patch to fix a misbehavior of hash indexes on
unlogged tables.

Julien Rouhaud sent in two more revisions of a patch to implement sorted append.

Nathan Bossart sent in three more revisions of a patch to enable specifying
multiple tables in VACUUM commands.

Nikita Glukhov sent in a patch to fix a circle bounding box inconsistency in the
GiST compress method and fix GiST ordering by distance for NULLs and NaNs.

Kyotaro HORIGUCHI sent in a patch to fix an issue where analyzeCTE is too strict
about typmods.

Amit Khandekar sent in another revision of a patch to enable UPDATEs of a
partition key which move the row to another partition.

Michaël Paquier sent in a patch to document new wait events.

Andres Freund sent in another revision of a patch to improve sys/catcache
performance.

Yura Sokolov sent in a patch to skip scanning a btree index if no pending
deleted pages exists.

Chen Huajun sent in another revision of a patch to make pg_rewind to not copy
useless WAL files.

Robert Haas sent in a patch to skip LSN checks during hash index scans.

Peter Eisentraut sent in a patch to expand the collation documentation and add
test cases for the new advanced ICU features.

Fabien COELHO sent in a patch to pgbench to use enum to store its meta commands.

Yura Sokolov sent in another revision of a patch to improve compactify_tuples.

Fabien COELHO sent in another revision of a patch to add server version
information to psql.

Tom Lane sent in a patch to make construct_[md_]array() handle empty arrays
correctly.

Andrew Dunstan sent in two more revisions of a patch to fix an infelicity in the
enum type.

Amit Kapila sent in another revision of a patch to change metapage usage for
B-tree and hash indexes.

Fabien COELHO sent in another revision of a patch to fix a pgbench regression
test failure.

Thomas Munro sent in a patch to improve the LDAP cleanup code in error paths,
log diagnostic messages if errors occur during LDAP auth, and add a regression
test to log an LDAP diagnostic message.

Mark Rofail sent in another revision of a patch to add foreign key arrays.

Tomas Vondra sent in another revision of a patch to make VACUUM and ANALYZE
agree on what reltuples means.

Erik Rijkers sent in a patch to improve some comments in prepunion.c.

Tomas Vondra sent in another revision of a patch to implement a generational
memory allocator.

Thomas Munro sent in another revision of a patch to use fallocate() to
preallocate "posix" DSM segments on Linux.