== PostgreSQL Weekly News - August 14 2017 ==

From: David Fetter <david(at)fetter(dot)org>
To: PostgreSQL Announce <pgsql-announce(at)postgresql(dot)org>
Subject: == PostgreSQL Weekly News - August 14 2017 ==
Date: 2017-08-14 01:51:44
Message-ID: 20170814015144.GB21618@fetter.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-announce

== PostgreSQL Weekly News - August 14 2017 ==

PostgreSQL 10 Beta 3 released. Test!
https://www.postgresql.org/about/news/1771/

Security releases 9.6.4, 9.5.8, 9.4.13, 9.3.18 and 9.2.22 are out. Please read
the announcement below and upgrade at the next available downtime.
https://www.postgresql.org/about/news/1772/

2ndQuadrant PostgreSQL Conference 2017 (2Q PGConf, for short) will be hosted on
November 6th & 7th in New York City, and November 9th in Chicago.
http://www.2qpgconf.com/

PGConf Local: Seattle Has been rescheduled to November 13 - 14, 2017.
Registration at https://www.pgconf.us/#Seattle2017

PGConf Local: Austin will be held December 4 - 5, 2017. Call for Papers is
now open at https://www.pgconf.us/conferences/Austin2017

== PostgreSQL Product News ==

PGroonga 1.2.3, a full text search platform for all languages, released.
http://groonga.org/en/blog/2017/07/03/pgroonga-1.2.3.html

JDBC 42.1.4 released
https://jdbc.postgresql.org/documentation/changelog.html#version_42.1.4

psqlODBC 09.06.0410 released.
https://odbc.postgresql.org/docs/release.html

psycopg2 2.7.3, a Python connector for PostgreSQL, released.
http://initd.org/psycopg/articles/2017/07/24/psycopg-273-released/

pg_repack 1.4.1, an extension for removing bloat, released.
http://reorg.github.io/pg_repack/

== PostgreSQL Jobs for August ==

http://archives.postgresql.org/pgsql-jobs/2017-08/

== PostgreSQL Local ==

PostgresOpen will occur September 6-8, 2017 in San Francisco.
https://2017.postgresopen.org/

PGBR2017 will take place in Porto Alegre, Rio Grande do Sul,
Brazil on September 14-16 2017.
https://pgbr.postgresql.org.br/2017/

PGDay.IT 2017 will take place in October 13th, in Milan, Italy.
http://pgday.it

PostgreSQL Conference Europe 2017 will be held on October 24-27, 2017 in the
Warsaw Marriott Hotel, in Warsaw, Poland.
https://2017.pgconf.eu/

PGConf Local: Seattle will be held November 13 - 14, 2017.
https://www.pgconf.us/#Seattle2017

PGDay Austin 2017 will be held on December 4 - 5, 2017.
https://pgdayaustin2017.postgresql.us

PGConf.ASIA 2017 will take place on December 4-6 2017 in Akihabara, Tokyo,
Japan.
http://www.pgconf.asia/EN/2017/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david(at)fetter(dot)org, German language
to pwn(at)pgug(dot)de, Italian language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Noah Misch pushed:

- Disclaim xmltable() support for non-UTF8 databases. The xmltable()
implementation mirrors xpath(), including its lack of character encoding
awareness.
https://git.postgresql.org/pg/commitdiff/b4a2eea030ba74ea84335c7d5bc999f693ffd9a4

- Fix vertical spanning in table "wait_event Description". Michael Paquier
Discussion:
https://postgr.es/m/CAB7nPqQr3KEQvXeuUNYcm7tDK2Fb9oLUQ8DU0+y0RZEoN_1_gg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/e88928c50dfe2623c899f82b54aad69da248ad07

Álvaro Herrera pushed:

- Fix replication origin-related race conditions. Similar to what was fixed in
commit 9915de6c1cb2 for replication slots, but this time it's related to
replication origins: DROP SUBSCRIPTION attempts to drop the replication
origin, but that fails if the replication worker process hasn't yet marked it
unused. This causes failures in the buildfarm: ERROR: could not drop
replication origin with OID 1, in use by PID 34069 Like the aforementioned
commit, fix by having the process running DROP SUBSCRIPTION sleep until the
worker marks the the replication origin struct as free. This uses a condition
variable on each replication origin shmem state struct, so that the session
trying to drop can sleep and expect to be awakened by the process keeping the
origin open. Also fix a SGML markup in the previous commit. Discussion:
https://postgr.es/m/20170808001433.rozlseaf4m2wkw3n@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/b2c95a3798ff39fc24d71b6655ddfe0e4cb3f378

- Fix inadequacies in recently added wait events In commit 9915de6c1cb2, we
introduced a new wait point for replication slots and incorrectly labelled it
as wait event PG_WAIT_LOCK. That's wrong, so invent an appropriate new wait
event instead, and document it properly. While at it, fix numerous other
problems in the vicinity: - two different walreceiver wait events were being
mixed up in a single wait event (which wasn't documented either); split it out
so that they can be distinguished, and document the new events properly. -
ParallelBitmapPopulate was documented but didn't exist. - ParallelBitmapScan
was not documented (I think this should be called "ParallelBitmapScanInit"
instead.) - Logical replication wait events weren't documented - various
symbols had been added in dartboard order in various places. Put them in
alphabetical order instead, as was originally intended. Discussion:
https://postgr.es/m/20170808181131.mu4fjepuh5m75cyq@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/030273b7ea468ed4b3073dfd1f2ad88e3129df6a

- Fix typo in comment.
https://git.postgresql.org/pg/commitdiff/f5d54ef97abdd1df3d6bfe0320a565ecce742abe

- Reword some unclear comments.
https://git.postgresql.org/pg/commitdiff/77d2c00af78ee12ae0d1cea34605f1e7af3f6d93

- Reword comment for clarity. Reported by Masahiko Sawada Discussion:
https://postgr.es/m/CAD21AoB+ycZ2z-4Ye=6MfQ_r0aV5r6cvVPw4kOyPdp6bHqQoBQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2336f842843e5bd26779692b39eaf0ef9d4d31da

Tom Lane pushed:

- Fix yet another race condition in recovery/t/001_stream_rep.pl. In commit
5c77690f6, we added polling in front of most of the get_slot_xmins calls in
001_stream_rep.pl, but today's results from buildfarm member nightjar show
that at least one more poll loop is needed. Proactively add a poll loop
before the next-to-last get_slot_xmins call as well. It may be that there is
no race condition there because the standby_2 server is shut down at that
point, but I'm quite tired of fighting with this test script. The empirical
evidence that it's safe, from the buildfarm, is no stronger than the evidence
for the other call that nightjar just proved unsafe. The only remaining
get_slot_xmins calls without wait_slot_xmins protection are the first two,
which should be OK since nothing has happened at that point. It's tempting to
ignore that special case and merge get_slot_xmins and wait_slot_xmins into a
single function. I didn't go that far though. Discussion:
https://postgr.es/m/18436.1502228036@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/4576a69354fa2efc1bafa50df1c104c1a80c64e5

- Fix datumSerialize infrastructure to not crash on non-varlena data. Commit
1efc7e538 did a poor job of emulating existing logic for touching Datums that
might be expanded-object pointers. It didn't check for typlen being -1 first,
which meant it could crash on fixed-length pass-by-ref values, and probably on
cstring values as well. It also didn't use DatumGetPointer before
VARATT_IS_EXTERNAL_EXPANDED, which while currently harmless is not according
to documentation nor prevailing style. I also think the lack of any
explanation as to why datumSerialize makes these particular nonobvious choices
is pretty awful, so fix that. Per report from Jarred Ward. Back-patch to 9.6
where this code came in. Discussion:
https://postgr.es/m/6F61E6D2-2F5E-4794-9479-A429BE1CEA4B@simple.com
https://git.postgresql.org/pg/commitdiff/9bf4068cc321a4d44ac54089ab651a49d89bb567

- Prevent passing down MAKELEVEL/MAKEFLAGS from non-GNU make to GNU make.
FreeBSD's make, for one, sets the MAKELEVEL environment variable when invoking
commands. In the special Makefile we provide to hand off control from a
non-GNU make to GNU make, this causes GNU make to think it is a child make
invocation rather than top-level. That interferes with the hack added in
commit dcae5facc to cause the temp-install tree to be made only by the
top-level invocation of gmake. Unset the variable to prevent that. Likewise
unset MAKEFLAGS, which FreeBSD's make also sets, and which could easily
confuse gmake. There are no reports of actual trouble from that, but it seems
better to be proactive. Back-patch to 9.5 where dcae5facc came in. Thomas
Munro, hacked a bit more by me Discussion:
https://postgr.es/m/CAEepm=1ueww35AXTkt1A3gyzZUqv5XCzh8RUNvJZAQAW=eOhVw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/a76200de8462aa0551dde8132c11d648d0ad6e99

- Fix handling of container types in find_composite_type_dependencies.
find_composite_type_dependencies correctly found columns that are of the
specified type, and columns that are of arrays of that type, but not columns
that are domains or ranges over the given type, its array type, etc. The most
general way to handle this seems to be to assume that any type that is
directly dependent on the specified type can be treated as a container type,
and processed recursively (allowing us to handle nested cases such as ranges
over domains over arrays ...). Since a type's array type already has such a
dependency, we can drop the existing special case for the array type. The
very similar logic in get_rels_with_domain was likewise a few bricks shy of a
load, as it supposed that a directly dependent type could *only* be a
sub-domain. This is already wrong for ranges over domains, and it'll someday
be wrong for arrays over domains. Add test cases illustrating the problems,
and back-patch to all supported branches. Discussion:
https://postgr.es/m/15268.1502309024@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/749c7c41701c62d96a56af1531e4f51e39173be3

- Remove pgbench's restriction on placement of -M switch. Previously the -M
switch had to appear before any switch that directly or indirectly specified a
benchmarking script. This was both confusing and inadequately documented, as
per gripe from Tatsuo Ishii. We can remove the restriction at the cost of
making an extra pass over the lists of SQL commands, which seems like a cheap
price (the string scans themselves likely cost much more). The change is just
to not extract parameters from the SQL commands until we have finished parsing
the switches and know the final value of -M. Per discussion, we'll treat this
as a low-grade bug fix and sneak it into v10, rather than holding it for v11.
Tom Lane, reviewed by Tatsuo Ishii and Fabien Coelho Discussion:
https://postgr.es/m/20170802.110328.1963639094551443169.t-ishii@sraoss.co.jp
Discussion: https://postgr.es/m/10208.1502465077@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/79681844297a9d499a10094a17e52b365afdd8bb

- Add regression tests exercising more code paths in nodeLimit.c. Perusal of
the code coverage report shows that the existing regression test cases for
LIMIT/OFFSET don't exercise the nodeLimit code paths involving backwards scan,
empty results, or null values of LIMIT/OFFSET. Improve the coverage.
https://git.postgresql.org/pg/commitdiff/3c8de95979008d67713429d858957c5c78c23d75

- Add regression tests exercising the non-hashed code paths in nodeSetop.c.
Perusal of the code coverage report shows that the existing regression test
cases for INTERSECT and EXCEPT seemingly all prefer the SETOP_HASHED
implementation. Add some test cases in which we force use of the SETOP_SORTED
mode.
https://git.postgresql.org/pg/commitdiff/6efca23cc039b6a0b25d2ebf4a22ab7363d17fcf

- Be more thorough about cleaning out gcov litter. At least on my machine, a
run with code coverage enabled produces some ".gcov" files whose names begin
with ".". "rm -f *.gcov" fails to match those, so they don't get cleaned up
by "make clean". Fix it.
https://git.postgresql.org/pg/commitdiff/d6ecad812f981e6ea611c1022ce7540830393a36

- Simplify fetch-slot-xmins logic in recovery TAP tests. Merge
wait_slot_xmins() into get_slot_xmins(). At this point the only place that
wasn't doing a wait was the initial-state test, and a wait there seems pretty
harmless. Michael Paquier Discussion:
https://postgr.es/m/CAB7nPqSp_SLQb2uU7am+sn4V3g1UKv8j3yZU385oAG1cG_BN9Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3043c1ddd13cd902d820d9aea353e60ce2d3d698

- Remove AtEOXact_CatCache(). The sole useful effect of this function, to check
that no catcache entries have positive refcounts at transaction end, has
really been obsolete since we introduced ResourceOwners in PG 8.1. We reduced
the checks to assertions years ago, so that the function was a complete no-op
in production builds. There have been previous discussions about removing it
entirely, but consensus up to now was that it had some small value as a
cross-check for bugs in the ResourceOwner logic. However, it now emerges that
it's possible to trigger these assertions if you hit an assert-enabled backend
with SIGTERM during a call to SearchCatCacheList, because that function
temporarily increases the refcounts of entries it's intending to add to a
catcache list construct. In a normal ERROR scenario, the extra refcounts are
cleaned up by SearchCatCacheList's PG_CATCH block; but in a FATAL exit we do a
transaction abort and exit without ever executing PG_CATCH handlers. There's
a case to be made that this is a generic hazard and we should consider
restructuring elog(FATAL) handling so that pending PG_CATCH handlers do get
run. That's pretty scary though: it could easily create more problems than it
solves. Preliminary stress testing by Andreas Seltenreich suggests that there
are not many live problems of this ilk, so we rejected that idea. There are
more-localized ways to fix the problem; the most principled one would be to
use PG_ENSURE_ERROR_CLEANUP instead of plain PG_TRY. But adding cycles to
SearchCatCacheList isn't very appealing. We could also weaken the assertions
in AtEOXact_CatCache in some more or less ad-hoc way, but that just makes its
raison d'etre even less compelling. In the end, the most reasonable solution
seems to be to just remove AtEOXact_CatCache altogether, on the grounds that
it's not worth trying to fix it. It hasn't found any bugs for us in many
years. Per report from Jeevan Chalke. Back-patch to all supported branches.
Discussion:
https://postgr.es/m/CAM2+6=VEE30YtRQCZX7_sCFsEpoUkFBV1gZazL70fqLn8rcvBA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/004a9702e062b5b6d61813f3fc8d71654d331d05

Peter Eisentraut pushed:

- doc: Add missing pieces to logical replication protocol doc. Reported-by:
Kyle Conroy <kyle(at)kyleconroy(dot)com>
https://git.postgresql.org/pg/commitdiff/13f03a001e4d841b6a27c0c9c3fe14e1fb2aad80

- Fix order of ICU_CFLAGS. It must be before CPPFLAGS so that an ICU
installation in a nonstandard path can take precedence over one in the system
path.
https://git.postgresql.org/pg/commitdiff/b83e54564ad0733f5382b20c04695ee9fb4cf451

- Reject use of ucol_strcollUTF8() before ICU 53. Various bugs can cause
crashes, so don't use that function before ICU 53. It will fall back to the
code path used for other encodings. Since we now tie the function
availability to an ICU version, we don't need the configure test anymore.
That also resolves the issue that the test result was previously hardcoded for
Windows. researched by Daniel Verite <daniel(at)manitou-mail(dot)org>, Peter
Geoghegan <pg(at)bowt(dot)ie>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> Discussion:
https://www.postgresql.org/message-id/flat/f1438ec6-22aa-4029-9a3b-26f79d330e72%40manitou-mail.org
https://git.postgresql.org/pg/commitdiff/d6391b03b3025372620925e5746e65c288a1e371

- Remove uses of "slave" in replication contexts. This affects mostly code
comments, some documentation, and tests. Official APIs already used
"standby".
https://git.postgresql.org/pg/commitdiff/a1ef920e27ba6ab3602aaf6d6751d8628fac1af8

- pg_upgrade: Clarify one message. Reported-by: Dennis Björklund
<db(at)zigo(dot)dhs(dot)org>
https://git.postgresql.org/pg/commitdiff/d4ede668d6f0ca9e5dd6def4ea1ccddc441c6073

- doc: Small wording improvement. Author: Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/22701a7ec66ffb3b62fae7f04ef36bc6ea21df52

- doc: Update description of rolreplication column. Since PostgreSQL 9.6,
rolreplication no longer determines whether a role can run pg_start_backup()
and pg_stop_backup(), so remove that. Add that this attribute determines
whether a role can create and drop replication slots. Reported-by: Fujii
Masao <masao(dot)fujii(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/fa65c8c73cb21ab3154db2f0f291227ba901c996

- doc: Add example for inet vs cidr difference. Reported-by: kes-kes(at)yandex(dot)ru
https://git.postgresql.org/pg/commitdiff/ee844bb42632521c89497a2845079770b32a934e

Robert Haas pushed:

- Remove incorrect assertion in clog.c. We must advance the oldest XID that can
be safely looked up in clog *before* truncating CLOG, and the oldest XID that
can't be reused *after* truncating CLOG. This assertion, and the accompanying
comment, are confused; remove them. Reported by Neha Sharma. Discussion:
http://postgr.es/m/CANiYTQumC3T=UMBMd1Hor=5XWZYuCEQBioL3ug0YtNQCMMT5wQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ec99dd5aee8b831760046d43098c2d1cf23157ed

- pgstatindex: Insert some casts to prevent overflow. This could cause hash
indexes to report greater than 100% free space. Ashutosh Sharma, reviewed by
Amit Kapila Discussion:
http://postgr.es/m/CAE9k0PnCKfg-ZK1CwGZJPF1yKcG2A=GUgC3BMdNMzLAXVOo4Eg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/0b7ba3d6474b8f58e74dba548886df3250805cdf

- Fix typo in comment. Etsuro Fujita Discussion:
http://postgr.es/m/5f794b91-67df-1ac6-8a4f-069f8e8e169d@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/e694010758772da1ac0f3355027fad0e47da4465

- Make some more improvements to parallel query documentation. Many places that
mentioned only Gather should also mention Gather Merge, or should be phrased
in a more neutral way. Be more clear about the fact that
max_parallel_workers_per_gather affects the number of workers the planner may
want to use. Fix a typo. Explain how Gather Merge works. Adjust wording
around parallel scans to be a bit more clear. Adjust wording around
parallel-restricted operations for the fact that uncorrelated subplans are no
longer restricted. Patch by me, reviewed by Erik Rijkers Discussion:
http://postgr.es/m/CA+TgmoZsTjgVGn=ei5ht-1qGFKy_m1VgB3d8+Rg304hz91N5ww@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/c1ef4e5cdb11cd562891f4ad2f30d1e3583a973d

- Improve the error message when creating an empty range partition. The
previous message didn't mention the name of the table or the bounds. Put the
table name in the primary error message and the bounds in the detail message.
Amit Langote, changed slightly by me. Suggestions on the exac phrasing from
Tom Lane, David G. Johnston, and Dean Rasheed. Discussion:
http://postgr.es/m/CA+Tgmoae6bpwVa-1BMaVcwvCCeOoJ5B9Q9-RHWo-1gJxfPBZ5Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/bb5d6e80b1387f0de58e55ac8e882f68ec6d4fcf

== Pending Patches ==

Piotr Stefaniak sent in a PoC patch to clarify the "sorry, too many clients
already" error.

Mark Rofail sent in another revision of a patch to implement Foreign Key Arrays.

Masahiko Sawada sent in a patch to enable pgbench to skip creating primary keys
after initialization.

Ashutosh Bapat and Thomas Munro traded patches to enable partition-wise joins
between declaratively partitioned tables.

Shubham Barai sent in another revision of a patch to add predicate locking in
GIN indexes.

Masahiko Sawada sent in another revision of a patch to improve the B-tree code.

Beena Emerson sent in two more revisions of a patch to add default partition
support for range partitions.

Amit Kapila sent in two more revisions of a patch to parallelize queries
containing initplans.

Masahiko Sawada sent in a patch to fix a typo in the login replication message.

Thomas Munro sent in another revision of a patch to log LDAP "diagnostic
messages."

Thomas Munro sent in another revision of a patch to implement causal reads.

Vaishnavi Prabakaran sent in another revision of a patch to add batch/pipelining
support for libpq.

Amit Khandekar sent in another revision of a patch to implement parallel append.

Michaël Paquier sent in a patch to enable creating backup history files for
backups taken from standbys.

Amit Langote sent in a patch to expand inheritance in partition bound order.

Oliver Ford sent in a patch to fix number-skipping in to_number.

Tom Lane sent in a patch to fix an issue where the PL/perl extension failed to
build on Windows.

Nicolas Thauvin and Ashutosh Bapat traded patches to fix an issue where foreign
tables privileges were not shown in information_schema.table_privileges.

Rushabh Lathia and Robert Haas traded patches to enable loading partitioned data
through the root.

Yura Sokolov sent in another revision of a patch to make a hash table for xip in
XidInMVCCSnapshot.

Michaël Paquier sent in a patch to simplify the TAP test around replication
slots.

Peter Eisentraut sent in a patch to rework and adjust how the coverage analysis
tools are run.

Amit Khandekar sent in another revision of a patch to enable UPDATEs of a
partition key.

Ashutosh Sharma sent in another revision of a patch to rewrite hash index scan
to work page at a time, remove redundant hash function _hash_step and do some
code cleanup, and improve locking startegy during VACUUM in Hash Index for
regular tables.

Thomas Munro sent in another revision of a patch to enable sharing record
typmods among backends.

Peter Eisentraut sent in a patch to add new contrib test suites.

Petr Jelínek sent in a patch to allow pg_dump -j on standby when dumping from
PG10+.

Dmitry Dolgov sent in another revision of a patch to enable generic type
subscripting.

Alik Khilazhev sent in another revision of a patch to enable generating zipfian
distributions in pgbench.

Fabien COELHO sent in another revision of a patch to enable pgbench to store
select results into variables.

Marko Tiikkaja sent in another revision of a patch to enable INSTEAD OF DELETE
triggers to modify the tuple for RETURNING.

Tom Lane sent in another revision of a patch to fix a bug that broke replication
to Windows in PostgreSQL 10.

Tomas Vondra sent in a patch to add multivariate histograms and MCV lists.

Tomas Vondra sent in a patch to create a generational memory allocator.

Browse pgsql-announce by date

  From Date Subject
Next Message D'Arcy Cain 2017-08-15 12:21:32 Release of PyGreSQL 5.0.4
Previous Message Daniele Varrazzo 2017-08-12 12:22:29 pg_repack 1.4.1 released