== PostgreSQL Weekly News - November 12 2017 ==

== PostgreSQL Weekly News - November 12 2017 ==

PostgreSQL security releases 10.1, 9.6.6, 9.5.10, 9.4.15, 9.3.20, and 9.2.24 are
out. Please read the announcement below and upgrade at the next available
downtime.
https://www.postgresql.org/about/news/1801/

PGConf.Brazil 2018 will take place in São Paulo, Brazil on August 3-4 2018. The
CfP will open soon.
http://pgconf.com.br

Nordic PGDay 2018 will be held in Oslo, Norway, at the Radisson Blu Hotel
Nydalen, on March 13, 2018. The CfP is open through December 31, 2017 at
https://2018.nordicpgday.org/cfp/

pgDay Paris 2018 will be held in Paris, France at the Espace Saint-Martin, on
March 15 2018. The CfP is open until December 31, 2017.
http://2018.pgday.paris/callforpapers/

== PostgreSQL Jobs for November ==

http://archives.postgresql.org/pgsql-jobs/2017-11/

== PostgreSQL Local ==

PGConf Local: Seattle will be held November 13 - 14, 2017.
https://www.pgconf.us/#Seattle2017

PGDay Australia 2017 will be held on November 17 in Melbourne.
http://2017.pgday.com.au/

PostgreSQL Session will take place November 17th, 2017, in Paris, France.
http://www.postgresql-sessions.org/en/9/start

PGDAY Brasilia 2017 will take place December 2, 2017. The CfP
is open at
http://www.pgdaybrasilia.com.br/submissao-de-palestra/

PGConf Local: Austin will be held December 4 - 5, 2017. Call for Papers is
now open at https://www.pgconf.us/conferences/Austin2017

PGConf.ASIA 2017 will take place on December 4-6 2017 in Akihabara, Tokyo,
Japan.
http://www.pgconf.asia/EN/2017/

FOSDEM PGDay 2018, a one day conference held before the main FOSDEM event will
be held in Brussels, Belgium, on Feb 2nd, 2018.
https://2018.fosdempgday.org/

Prague PostgreSQL Developer Day 2018 (P2D2 2018) is a two-day
conference that will be held on February 14-15 2018 in Prague, Czech Republic.
The CfP is open until January 5, 2018 at https://p2d2.cz/callforpapers
http://www.p2d2.cz/

PGConf India 2018 will be on February 22-23, 2018 in Bengaluru, Karnataka.
Proposals are due via https://goo.gl/forms/F9hRjOIsaNasVOAz2 by October 31st, 2017.
http://pgconf.in/

PostgreSQL(at)SCaLE is a two day, two track event which takes place on
March 8-9, 2018, at Pasadena Convention Center, as part of SCaLE 16X.
The CfP is open through October 31, 2017 at
http://www.socallinuxexpo.org/scale/16x/cfp

PGConf APAC 2018 will be held in Singapore March 22-24, 2018. The CfP is open
at http://2018.pgconfapac.org/cfp through December 4, 2017.
http://2018.pgconfapac.org/

The German-speaking PostgreSQL Conference 2018 will take place on April 13th,
2018 in Berlin. The CfP is open until January 09, 2018 at
http://2018.pgconf.de/de/callforpapers.html and the conference site is at
http://2018.pgconf.de/

PGCon 2018 will take place in Ottawa on May 29 - June 2018. The CfP goes out
soon.
https://www.pgcon.org/2018/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm EST5EDT. Please send English
language ones to david(at)fetter(dot)org, German language to pwn(at)pgug(dot)de, Italian
language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Release notes for 10.1, 9.6.6, 9.5.10, 9.4.15, 9.3.20, 9.2.24. In the v10
branch, also back-patch the effects of 1ff01b390 and c29c57890 on these files,
to reduce future maintenance issues. (I'd do it further back, except that the
9.X branches differ anyway due to xlog-to-wal link tag renaming.)
https://git.postgresql.org/pg/commitdiff/b35b185bf705c4dbaf21198c81b3d85f4a96804a

- Fix version numbering foulups exposed by 10.1. configure computed
PG_VERSION_NUM incorrectly. (Coulda sworn I tested that logic back when, but
it had an obvious thinko.) pg_upgrade had not been taught about the new
dispensation with just one part in the major version number. Both things
accidentally failed to fail with 10.0, but with 10.1 we got the wrong results.
Per buildfarm.
https://git.postgresql.org/pg/commitdiff/d0c80c17f1a6d0b93d2ca14fe47d83b131ce9108

- Fix unportable usage of <ctype.h> functions. isdigit(), isspace(), etc are
likely to give surprising results if passed a signed char. We should always
cast the argument to unsigned char to avoid that. Error in commit 63d6b97fd,
found by buildfarm member gaur. Back-patch to 9.3, like that commit.
https://git.postgresql.org/pg/commitdiff/d1f9ac5b100dbc4da02f0f209a2e7730bd5e83e9

- Fix unportable spelling of int64 constant. Per buildfarm member pademelon.
https://git.postgresql.org/pg/commitdiff/92a1834dd88e56e823ac6641313a2f077a8af72e

- Fix two violations of the ResourceOwnerEnlarge/Remember protocol. The point
of having separate ResourceOwnerEnlargeFoo and ResourceOwnerRememberFoo
functions is so that resource allocation can happen in between. Doing it in
some other order is just wrong. OpenTemporaryFile() did open(), enlarge,
remember, which would leak the open file if the enlarge step ran out of
memory. Because fd.c has its own layer of resource-remembering, the
consequences look like they'd be limited to an intratransaction FD leak, but
it's still not good. IncrBufferRefCount() did enlarge, remember,
incr-refcount, which would blow up if the incr-refcount step ever failed. It
was safe enough when written, but since the introduction of
PrivateRefCountHash, I think the assumption that no error could happen there
is pretty shaky. The odds of real problems from either bug are probably
small, but still, back-patch to supported branches. Thomas Munro and Tom
Lane, per a comment from Andres Freund
https://git.postgresql.org/pg/commitdiff/c5269472ea9bb4a6fbb8a0510f7d676d725933ab

- Doc: fix erroneous example. The grammar requires these options to appear the
other way 'round. jotpe(at)posteo(dot)de Discussion:
https://postgr.es/m/78933bd0-45ce-690e-b832-a328dd1a5567@posteo.de
https://git.postgresql.org/pg/commitdiff/bd65e0c62486e6108a7dc824f918754a13072f7a

- Allow --with-bonjour to work with non-macOS implementations of Bonjour. On
macOS the relevant functions require no special library, but elsewhere we need
to pull in libdns_sd. Back-patch to supported branches. No docs change since
the docs do not suggest that this is a Mac-only feature. Luke Lonergan
Discussion:
https://postgr.es/m/2D8331C5-D64F-44C1-8717-63EDC6EAF7EB@brightforge.com
https://git.postgresql.org/pg/commitdiff/9b9cb3c4534d717c1c95758670198ebbf8a20af2

- Last-minute updates for release notes. Security: CVE-2017-12172,
CVE-2017-15098, CVE-2017-15099
https://git.postgresql.org/pg/commitdiff/92d830f4bff643953a09563abaa106af42625207

- Add tests for json{b}_populate_recordset() crash case. The problem reported
as CVE-2017-15098 was already resolved in HEAD by commit 37a795a60, but let's
add the relevant test cases anyway. Michael Paquier and Tom Lane, per a
report from David Rowley. Security: CVE-2017-15098
https://git.postgresql.org/pg/commitdiff/b574228715f0fd77ed1f4f084603cff9e757e992

- Revert "Allow --with-bonjour to work with non-macOS implementations of
Bonjour.". Upon further review, our Bonjour code doesn't actually work with
the Avahi not-too-compatible compatibility library. While you can get it to
work on non-macOS platforms if you link to Apple's own mDNSResponder code,
there don't seem to be many people who care about that. Leaving in the
AC_SEARCH_LIBS call seems more likely to encourage people to build broken
configurations than to do anything very useful. Hence, remove the
AC_SEARCH_LIBS call and put in a warning comment instead. Discussion:
https://postgr.es/m/2D8331C5-D64F-44C1-8717-63EDC6EAF7EB@brightforge.com
https://git.postgresql.org/pg/commitdiff/20d9adab60754ac71b0b500c91c45e12e940b3ce

- Fix bogus logic for checking executables' versions within pg_upgrade.
Somebody messed up a refactoring here. As it stood, we'd check pg_ctl's
--version output twice for each cluster. Worse, the first check for the new
cluster's version happened before we'd done any validate_exec checks there,
breaking the check ordering the code intended. A. Akenteva Discussion:
https://postgr.es/m/f9266a85d918a3cf3a386b5148aee666@postgrespro.ru
https://git.postgresql.org/pg/commitdiff/9be95ef156e7c2ae0924300acddd483504fa33b3

- Fix typo in ALTER SYSTEM output. The header comment written into
postgresql.auto.conf by ALTER SYSTEM should match what initdb put there
originally. Feike Steenbergen Discussion:
https://postgr.es/m/CAK_s-G0KcKdO=0hqZkwb3s+tqZuuHwWqmF5BDsmoO9FtX75r0g@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/6c3a7ba5bb0f960ed412b1c36e815f53347b3d79

- Restrict lo_import()/lo_export() via SQL permissions not hard-wired checks.
While it's generally unwise to give permissions on these functions to anyone
but a superuser, we've been moving away from hard-wired permission checks
inside functions in favor of using the SQL permission system to control
access. Bring lo_import() and lo_export() into compliance with that approach.
In particular, this removes the manual configuration option
ALLOW_DANGEROUS_LO_FUNCTIONS. That dates back to 1999 (commit 4cd4a54c8);
it's unlikely anyone has used it in many years. Moreover, if you really want
such behavior, now you can get it with GRANT ... TO PUBLIC instead. Michael
Paquier Discussion:
https://postgr.es/m/CAB7nPqRHmNOYbETnc_2EjsuzSM00Z+BWKv9sy6tnvSd5gWT_JA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/5ecc0d738e5864848bbc2d1d97e56d5846624ba2

- Refactor permissions checks for large objects. Up to now, ACL checks for
large objects happened at the level of the SQL-callable functions, which led
to CVE-2017-7548 because of a missing check. Push them down to be enforced in
inv_api.c as much as possible, in hopes of preventing future bugs. This does
have the effect of moving read and write permission errors to happen at
lo_open time not loread or lowrite time, but that seems acceptable. Michael
Paquier and Tom Lane Discussion:
https://postgr.es/m/CAB7nPqRHmNOYbETnc_2EjsuzSM00Z+BWKv9sy6tnvSd5gWT_JA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ae20b23a9e7029f31ee902da08a464d968319f56

- Tighten test in contrib/bloom/t/001_wal.pl. Make bloom WAL test compare psql
output text, not just result codes; this was evidently the intent all along,
but it was mis-coded. In passing, make sure we will notice any failure in
setup steps. Alexander Korotkov, reviewed by Michael Paquier and Masahiko
Sawada Discussion:
https://postgr.es/m/CAPpHfdtohPdQ9rc5mdWjxq+3VsBNw534KV_5O65dTQrSdVJNgw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7e60e678615b1f803ac73faee71cca79ec310d1d

Noah Misch pushed:

- Add a temp-install prerequisite to "check"-like targets not having one.
Makefile.global assigns this prerequisite to every target named "check", but
similar targets must mention it explicitly. Affected targets failed, tested
$PATH binaries, or tested a stale temporary installation. The
src/test/modules examples worked properly when called as "make -C
src/test/modules/$FOO check", but "make -j" allowed the test to start before
the temporary installation was in place. Back-patch to 9.5, where commit
dcae5faccab64776376d354decda0017c648bb53 introduced the shared temp-install.
https://git.postgresql.org/pg/commitdiff/c66b438db62748000700c9b90b585e756dd54141

- start-scripts: switch to $PGUSER before opening $PGLOG. By default, $PGUSER
has permission to unlink $PGLOG. If $PGUSER replaces $PGLOG with a symbolic
link, the server will corrupt the link-targeted file by appending log
messages. Since these scripts open $PGLOG as root, the attack works
regardless of target file ownership. "make install" does not install these
scripts anywhere. Users having manually installed them in the past should
repeat that process to acquire this fix. Most script users have $PGLOG
writable to root only, located in $PGDATA. Just before updating one of these
scripts, such users should rename $PGLOG to $PGLOG.old. The script will then
recreate $PGLOG with proper ownership. Reviewed by Peter Eisentraut.
Reported by Antoine Scemama. Security: CVE-2017-12172
https://git.postgresql.org/pg/commitdiff/dfc015dcf46c1996bd7ed5866e9e045d258604b3

- Ignore XML declaration in xpath_internal(), for UTF8 databases. When a value
contained an XML declaration naming some other encoding, this function
interpreted UTF8 bytes as the named encoding, yielding mojibake. xml_parse()
already has similar logic. This would be necessary but not sufficient for
non-UTF8 databases, so preserve behavior there until the xpath facility can
support such databases comprehensively. Back-patch to 9.3 (all supported
versions). Pavel Stehule and Noah Misch Discussion:
https://postgr.es/m/CAFj8pRC-dM=tT=QkGi+Achkm+gwPmjyOayGuUfXVumCxkDgYWg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2918fcedbf2b2adab688a7849ecce4556ef912ac

- Fix previous commit's test, for non-UTF8 databases with non-XML builds. To
ensure stable output, catch one more configuration-specific error. Back-patch
to 9.3, like the commit that added the test.
https://git.postgresql.org/pg/commitdiff/4b865aee2582292a42a8e58247a41b46f5aa8a82

- Make connect/test1 independent of localhost IPv6. Since commit
868898739a8da9ab74c105b8349b7b5c711f265a, it has assumed "localhost" resolves
to both ::1 and 127.0.0.1. We gain nothing from that assumption, and it does
not hold in a default installation of Red Hat Enterprise Linux 5. Back-patch
to 9.3 (all supported versions).
https://git.postgresql.org/pg/commitdiff/34baf8a00b018caf7269134cf9b461266e66d9a7

- Add post-2010 ecpg tests to checktcp. This suite had been a proper superset
of the regular ecpg test suite, but the three newest tests didn't reach it.
To make this less likely to recur, delete the extra schedule file and pass the
TCP-specific test on the command line. Back-patch to 9.3 (all supported
versions).
https://git.postgresql.org/pg/commitdiff/0b7e76eb2b142d0b4a2a831e7fa1fac44820f52c

Simon Riggs pushed:

- Exclude pg_internal.init from BASE_BACKUP. Add docs to explain this for other
backup mechanisms Author: David Steele <david(at)pgmasters(dot)net> Reviewed-by: Petr
Jelinek <petr(dot)jelinek(at)2ndQuadrant(dot)com> et al
https://git.postgresql.org/pg/commitdiff/98267ee83e068afa8bd85c8cc710ab2144e711fa

- Remove secondary checkpoint. Previously server reserved WAL for last two
checkpoints, which used too much disk space for small servers. Bumps
PG_CONTROL_VERSION Author: Simon Riggs <simon(at)2ndQuadrant(dot)com> Reviewed-by:
Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/4b0d28de06b28e57c540fca458e4853854fbeaf8

Robert Haas pushed:

- Fix typo in comment. Masahiko Sawada Discussion:
http://postgr.es/m/CAD21AoDrf5AOpZ-mX-j6O=zFNFfKaTdHkv3o1X2eSs2nBXALug@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/698e5254377baffb95774d47872100997f135cba

- Fix incorrect comment. Etsuro Fujita Discussion:
http://postgr.es/m/5A05728E.4050009@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/b9941d3468505aea8bfdd74840b753ed27b9d29f

- Account for the effect of lossy pages when costing bitmap scans. Dilip Kumar,
reviewed by Alexander Kumenkov, Amul Sul, and me. Some final adjustments by
me. Discussion:
http://postgr.es/m/CAFiTN-sYtqUOXQ4SpuhTv0Z9gD0si3YxZGv_PQAAMX8qbOotcg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/5edc63bda68a77c4d38f0cbeae1c4271f9ef4100

- Add hash partitioning. Hash partitioning is useful when you want to partition
a growing data set evenly. This can be useful to keep table sizes reasonable,
which makes maintenance operations such as VACUUM faster, or to enable
partition-wise join. At present, we still depend on constraint exclusion for
partitioning pruning, and the shape of the partition constraints for hash
partitioning is such that that doesn't work. Work is underway to fix that,
which should both improve performance and make partitioning pruning work with
hash partitioning. Amul Sul, reviewed and tested by Dilip Kumar, Ashutosh
Bapat, Yugo Nagata, Rajkumar Raghuwanshi, Jesper Pedersen, and by me. A few
final tweaks also by me. Discussion:
http://postgr.es/m/CAAJ_b96fhpJAP=ALbETmeLk1Uni_GFZD938zgenhF49qgDTjaQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/1aba8e651ac3e37e1d2d875842de1e0ed22a651e

Peter Eisentraut pushed:

- Expand empty end tag.
https://git.postgresql.org/pg/commitdiff/6e1e4c0d2f757677b03b6f164487a582c7dfab33

- Put markup in the right place.
https://git.postgresql.org/pg/commitdiff/4497f2f3b30fa5cd48898033c351bfcf01ce73e2

- Change TRUE/FALSE to true/false. The lower case spellings are C and C++
standard and are used in most parts of the PostgreSQL sources. The upper case
spellings are only used in some files/modules. So standardize on the standard
spellings. The APIs for ICU, Perl, and Windows define their own TRUE and
FALSE, so those are left as is when using those APIs. In code comments, we
use the lower-case spelling for the C concepts and keep the upper-case
spelling for the SQL concepts. Reviewed-by: Michael Paquier
<michael(dot)paquier(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/2eb4a831e5fb5d8fc17e13aea56e04af3efe27b4

- Remove junk left from DSSSL to XSL conversion.
https://git.postgresql.org/pg/commitdiff/e7397f015c9589f95f5f5b48d7a274b2f1628971

- Add -wnet to SP invocations. This causes a warning when accidentally
backpatching an XML-style empty-element tag like <xref linkend="abc"/>.
https://git.postgresql.org/pg/commitdiff/9a8d3c4eeaf34966056a41a263c6e2ca4d5e4012

- Add some const decorations to prototypes. Reviewed-by: Fabien COELHO
<coelho(at)cri(dot)ensmp(dot)fr>
https://git.postgresql.org/pg/commitdiff/0e1539ba0d0a43de06c6e0572a565e73b9472538

- Fix some null pointer dereferences in LDAP auth code. An LDAP URL without a
host name such as "ldap://" or without a base DN such as "ldap://localhost"
would cause a crash when reading pg_hba.conf. If no binddn is configured, an
error message might end up trying to print a null pointer, which could crash
on some platforms. Author: Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com>
Reviewed-by: Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/0c98d0dd5c85ce0c8485ae1a8351a26b83c4338b

Dean Rasheed pushed:

- Always require SELECT permission for ON CONFLICT DO UPDATE. The update path
of an INSERT ... ON CONFLICT DO UPDATE requires SELECT permission on the
columns of the arbiter index, but it failed to check for that in the case of
an arbiter specified by constraint name. In addition, for a table with row
level security enabled, it failed to check updated rows against the table's
SELECT policies when the update path was taken (regardless of how the arbiter
index was specified). Backpatch to 9.5 where ON CONFLICT DO UPDATE and RLS
were introduced. Security: CVE-2017-15099
https://git.postgresql.org/pg/commitdiff/87b2ebd352c4afe1ded0841604b59a3afbae97d1

== Pending Patches ==

Edmund Horner sent in a patch to add some tab completion to the SELECT target
list in psql.

Ashutosh Bapat sent in a patch to improve \d+ output of a partitioned table and
separate default partition from rest of the partitions in psql.

Noah Misch sent in a patch to fix a race to build pg_isolation_regress in "make
-j check-world".

David Rowley sent in a patch to remove useless DISTINCT clauses.

Thomas Munro sent in a patch atop the statement-level rollback patch to document
it better.

Konstantin Knizhnik sent in another revision of a patch to optimize secondary
index lookups.

Raúl Marín Rodríguez sent in three more revisions of a patch to add pow() to
pgbench.

Asim Praveen sent in two more revisions of a patch to change incorrect calls to
PageGetLSN to BufferGetLSNAtomic and assert that locks are properly held in
PageGetLSN.

Kyotaro HORIGUCHI and Pavel Stěhule traded patches to add default namespaces for
XPath expressions.

Amit Langote and David Rowley traded patches to prune partitions faster.

Haribabu Kommi sent in another revision of a patch to create the infrastructure
for pluggable storage.

Nathan Bossart sent in two more revisions of a patch to add new logging for
VACUUM and ANALYZE.

Haribabu Kommi sent in another revision of a patch to add a pg_stat_wal_write
statistics view.

Haribabu Kommi sent in two more revisions of a patch to refactor handling of
database attributes between pg_dump and pg_dumpall.

Tom Lane sent in a patch to speed up compactify_tuples, this time without a sort
step.

Alexander Korotkov sent in two more revisions of a patch to fix the bloom WAL
tap test.

Thomas Munro sent in a patch to add planning counters to pg_stat_statements.

Michaël Paquier and Fabrízio de Royes Mello traded patches to add session_start
and session_end hooks.

Dmitry Dolgov and Artur Zakirov traded patches to add generic subscripting,
implementing those for arrays and JSON with same.

Masahiko Sawada sent in a patch to remove a duplicate setting in
test/recovery/Makefile.

Kyotaro HORIGUCHI sent in another revision of a patch to restrict the maximum
keep segments by repslots.

Amit Khandekar sent in two more revisions of a patch to enable UPDATEs of a
partition key to move tuples to the appropriate partition.

Etsuro Fujita sent in a patch to reorder header files in alphabetical order.

Robert Haas sent in another revision of a patch to implement parallel append.

Amit Kapila sent in another revision of a patch to ensure that parallel paths
include tlist cost.

Beena Emerson sent in another revision of a patch to implement runtime partiton
pruning.

Pavel Stěhule sent in two more revisions of a patch to add \graw to psql.

Michaël Paquier sent in a patch to remove ALLOW_DANGEROUS_LO_FUNCTIONS for
LO-related superuser checks, replace superuser checks of large object
import/export by ACL checks, and move ACL checks for large objects when opening
them.

Huong Dangminh sent in a patch to fix an issue where user-defined data types
weren't working right under logical replication.

Mark Rofail sent in another revision of a patch to implement foreign key arrays.

David Rowley sent in a patch to add a mention of CREATE STATISTICS to the event
trigger docs.

Robert Haas sent in another revision of a patch to speed up processing at Gather
nodes.

Graham Leggett sent in a patch to log SSL certificate verification errors.

Robert Haas and Amit Kapila traded patches to parallelize queries containing
initplans.

Pavel Stěhule sent in another revision of a patch to add a SERVER_VERSION
variable to psql.

Robert Haas sent in a patch to ensure that GatherMerge pushes tlists down.

Thomas Munro sent in a patch to add a parallel_leader_participation GUC.

Andrey Borodin sent in another revision of a patch to add GiST VACUUM.