== PostgreSQL Weekly News - July 12 2015 ==

== PostgreSQL Weekly News - July 12 2015 ==

PGDay.IT 2015 will take place in Prato on October 23, 2015. The
International Call For Papers is open until August 8.
http://pgday.it

The second official Russian PostgreSQL conference will be held on July
15-17, 2015 in St. Petersburg, Russia. A PostgreSQLRussia meetup for
new users will be held July 17, same venue.
http://PgDay.ru
http://PostgreSQLRussia.org

Rubens Souza from 2ndQuadrant Italia will lead a meetup on "How to
install PostgreSQL on a Raspberry PI" on Thursday, July 23 in Prato.
http://goo.gl/YYpsy7

== PostgreSQL Product News ==

Pervasync 6.0.2, a synchronization framework that supports PosrgreSQL,
released.
http://www.pervasync.com

pgBadger 7.1, a parallel PostgreSQL log analyzer written in Perl, released:
http://dalibo.github.io/pgbadger/
Development:
https://github.com/dalibo/pgbadger/

PipelineDB 0.7.7, an open-source, relational streaming-SQL database
based on PostgreSQL, released.
http://www.pipelinedb.com/

== PostgreSQL Jobs for July ==

http://archives.postgresql.org/pgsql-jobs/2015-07/

== PostgreSQL Local ==

PGDay Campinas 2015 will take place in Campinas on August 7.
http://pgdaycampinas.com.br/english/

PostgresOpen 2015 will being held in Dallas, Texas September 16-18.
http://2015.postgresopen.org/

PostgreSQL Session #7, will be held September 24th, 2015 in Paris,
France,
http://www.postgresql-sessions.org/7/about

PostgreSQL Conference Europe 2015 will be held on October 27-30 in the
Vienna Marriott Hotel, in Vienna, Austria. The CfP is open until
August 7.
http://2015.pgconf.eu/

PGConf Silicon Valley 2015 is November 17-18 at the South San
Francisco Convention Center.
http://www.pgconfsv.com

PGBR2015 (The Brazilian PostgreSQL Conference) will take place in Porto
Alegre, Rio Grande do Sul, on November 18, 19 and 20.
The CfP is open through July 15.
http://pgbr.postgresql.org.br/2015/en/#call-for-papers

PostgreSQL Conference Europe 2015 will be held on October 27-30 in the
Vienna Marriott Hotel, in Vienna, Austria. The CfP is open until
August 7.
http://2015.pgconf.eu/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david(at)fetter(dot)org, German language
to pwn(at)pgug(dot)de, Italian language to pwn(at)itpug(dot)org(dot) Spanish language
to pwn(at)arpug(dot)com(dot)ar(dot)

== Applied Patches ==

Andres Freund pushed:

- Fix logical decoding bug leading to inefficient reopening of files.
When spilling transaction data to disk a simple typo caused the
output file to be closed and reopened for every serialized change.
That happens to not have a huge impact on linux, which is why it
probably wasn't noticed so far, but on windows that appears to
trigger actual disk writes after every change. Not fun. The bug
fortunately does not have any impact besides speed. A change could
end up being in the wrong segment (last instead of next), but since
we read all files to the end, that's just ugly, not really
problematic. It's not a problem to upgrade, since transaction spill
files do not persist across restarts. Bug: #13484 Reported-By:
Olivier Gosseaume Discussion:
20150703090217(dot)1190(dot)63940(at)wrigleys(dot)postgresql(dot)org Backpatch to 9.4,
where logical decoding was added.
http://git.postgresql.org/pg/commitdiff/b2f6f749c7a5936adbb555e248e8e4df35c00a4a

- Fix pg_recvlogical not to fsync output when it's a tty or pipe. The
previous coding tried to handle possible failures when fsyncing a
tty or pipe fd by accepting EINVAL - but apparently some platforms
(windows, OSX) don't reliably return that. So instead check whether
the output fd refers to a pipe or a tty when opening it.
Reported-By: Olivier Gosseaume, Marko Tiikkaja Discussion:
559AF98B(dot)3050901(at)joh(dot)to Backpatch to 9.4, where pg_recvlogical was
added.
http://git.postgresql.org/pg/commitdiff/5c0de384d2ceceb07e77e1368e07868244be6762

- Add psql PROMPT variable showing the pid of the connected to
backend. The substitution for the pid is %p. Author: Julien
Rouhaud Discussion:
116262CF971C844FB6E793F8809B51C6E99D48(at)BPXM02GP(dot)gisp(dot)nec(dot)co(dot)jp
http://git.postgresql.org/pg/commitdiff/275f05c990c46f8dfe3cb46a3279521bda9e9e27

- Refer to %p in the psql docs as 'process ID' not 'pid'. Per Tom
Lane.
http://git.postgresql.org/pg/commitdiff/4af04f96bc9f604a57bf829469b26d8513fd6401

- Optionally don't error out due to preexisting slots in commandline
utilities. pg_receivexlog and pg_recvlogical error out when
--create-slot is specified and a slot with the same name already
exists. In some cases, especially with pg_receivexlog, that's rather
annoying and requires additional scripting. Backpatch to 9.5 as
slot control functions have newly been added to pg_receivexlog, and
there doesn't seem much point leaving it in a less useful state.
Discussion: 20150619144755(dot)GG29350(at)alap3(dot)anarazel(dot)de
http://git.postgresql.org/pg/commitdiff/ff27db5dd2fc096d89d3f995d3f650ec6d3bc147

- For consistency add a pfree to ON CONFLICT set_plan_refs code.
Backpatch to 9.5 where ON CONFLICT was introduced. Author: Peter
Geoghegan
http://git.postgresql.org/pg/commitdiff/3ed26e5f87a90bedaa3d7feb9e197e0d9f3fb252

Fujii Masao pushed:

- Add tab-completion for psql meta-commands. Based on the original
code from David Christensen, modified by me.
http://git.postgresql.org/pg/commitdiff/c81c956477c270c4dd816c2f659743ebcae02b77

- Fix incorrect path in pg_regress log messages. Back-patch to 9.5
where the bug was introduced. David Christensen
http://git.postgresql.org/pg/commitdiff/61fc420b5710036f3124a44c26a3664a9cc7cdcf

- Make wal_compression PGC_SUSET rather than PGC_USERSET. When
enabling wal_compression, there is a risk to leak data similarly to
the BREACH and CRIME attacks on SSL where the compression ratio of a
full page image gives a hint of what is the existing data of this
page. This vulnerability is quite cumbersome to exploit in
practice, but doable. So this patch makes wal_compression PGC_SUSET
in order to prevent non-superusers from enabling it and exploiting
the vulnerability while DBA thinks the risk very seriously and
disables it in postgresql.conf. Back-patch to 9.5 where
wal_compression was introduced.
http://git.postgresql.org/pg/commitdiff/c2e5f4d1c16faa84a19906706481cf149769f320

- Fix obsolete comment regarding NOTICE message level. By default
NOTICE message is not sent to server log because the default value
of log_min_messages is WARNING since 8.4. Pavel Stehule
http://git.postgresql.org/pg/commitdiff/6ba365aa4621b0e4c4c0920cbdf56348875a46a2

Heikki Linnakangas pushed:

- Improve handling of out-of-memory in libpq. If an allocation fails
in the main message handling loop, pqParseInput3 or pqParseInput2,
it should not be treated as "not enough data available yet".
Otherwise libpq will wait indefinitely for more data to arrive from
the server, and gets stuck forever. This isn't a complete fix -
getParamDescriptions and getCopyStart still have the same issue, but
it's a step in the right direction. Michael Paquier and me.
Backpatch to all supported versions.
http://git.postgresql.org/pg/commitdiff/414bef30bfab20451e15fe799642b52166db8c34

- Move pthread-tests earlier in the autoconf script. On some Linux
systems, "-lrt" exposed pthread-functions, so that linking with -lrt
was seemingly enough to make a program that uses pthreads to work.
However, when linking libpq, the dependency to libpthread was not
marked correctly, so that when an executable was linked with -lpq
but without -pthread, you got errors about undefined pthread_*
functions from libpq. To fix, test for the flags required to use
pthreads earlier in the autoconf script, before checking any other
libraries. This should fix the failure on buildfarm member
shearwater. gharial is also failing; hopefully this fixes that too
although the failure looks somewhat different.
http://git.postgresql.org/pg/commitdiff/3b14a17c8e60e8ae9227b9533667743508056c35

- Use AS_IF rather than plain shell "if" in pthread-check. Autoconf
generates additional code for the first AC_CHECK_HEADERS call in the
script. If the first call is within an if-block, the additional code
is put inside the if-block too, even though it is needed by
subsequent AC_CHECK_HEADERS checks and should always be executed.
When I moved the pthread-related checks earlier in the script, the
pthread.h test inside the block became the very first
AC_CHECK_HEADERS call in the script, triggering that problem. To
fix, use AS_IF instead of plain shell if. AS_IF knows about that
issue, and makes sure the additional code is always executed. To be
completely safe from this issue (and others), we should always be
using AS_IF instead of plain if, but that seems like excessive
caution given that this is the first time we have trouble like this.
Plain if-then is more readable than AS_IF. This should fix
compilation with --disable-thread-safety, and hopefully the
buildfarm failure on forgmouth, related to mingw standard headers,
too. I backpatched the previous fixes to 9.5, but it's starting to
look like these changes are too fiddly to backpatch, so commit this
to master only, and revert all the pthread-related configure changes
in 9.5.
http://git.postgresql.org/pg/commitdiff/01051a9879fcd353eaf0d3788a911e774b52798c

- Improve logging of TAP tests. Create a log file for each test run.
Stdout and stderr of the test script, as well as any subprocesses
run as part of the test, are redirected to the log file. This makes
it a lot easier to debug test failures. Also print the test output
(ok 12 - ... messages) to the log file, and the command line of any
external programs executed with the system_or_bail and run_log
functions. This makes it a lot easier to debug failing tests.
Modify some of the pg_ctl and other command invocations to not use
'silent' or 'quiet' options, and don't redirect output to /dev/null,
so that you get all the information in the log instead. In the
passing, construct some command lines in a way that works if
$tempdir contains quote-characters. I haven't systematically gone
through all of them or tested that, so I don't know if this is
enough to make that work. pg_rewind tests had a custom mechanism
for creating a similar log file. Use the new generic facility
instead. Michael Paquier and me.
http://git.postgresql.org/pg/commitdiff/1ea06203b82b98b5098808667f6ba652181ef5b2

- Fix another broken link in documentation. Tom fixed another one of
these in commit 7f32dbcd, but there was another almost identical one
in libpq docs. Per his comment: HP's web server has apparently
become case-sensitive sometime recently. Per bug #13479 from Daniel
Abraham. Corrected link identified by Alvaro.
http://git.postgresql.org/pg/commitdiff/aaec6a6d37b664199fd7744b976a7dc912ae000a

- Use --debug flag in "remote" pg_rewind regression tests. Gives more
information in the log, to debug possible failures.
http://git.postgresql.org/pg/commitdiff/23b8928829038ef3fba5a04e4f2707c6034464c4

- Replace our hacked version of ax_pthread.m4 with latest upstream
version. Our version was different from the upstream version in
that we tried to use all possible pthread-related flags that the
compiler accepts, rather than just the first one that works. That
change was made in commit e48322a6d6cfce1ec52ab303441df329ddbc04d1,
to work-around a bug affecting GCC versions 3.2 and below
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=8888), although we
didn't realize that it was a GCC bug at the time. We hardly care
about that old GCC versions anymore, so we no longer need that
workaround. This fixes the macro for compilers that print warnings
with the chosen flags. That's pretty annoying on its own right, but
it also inconspicuously disabled thread-safety, because we refused
to use any pthread-related flags if the compiler produced warnings.
Max Filippov reported that problem when linking with uClibc and
OpenSSL. The warnings-check was added because the workaround for the
GCC bug caused warnings otherwise, so it's no longer needed either.
We can just use the upstream version as is. If you really want to
compile with GCC version 3.2 or older, you can still work-around it
manually by setting PTHREAD_CFLAGS="-pthread -lpthread" manually on
the configure command line. Backpatch to 9.5. I don't want to
unnecessarily rock the boat on stable branches, but 9.5 seems like
fair game.
http://git.postgresql.org/pg/commitdiff/e97af6c8bfc80d084bca5bf41f036de944b63efe

- Copy-edit the docs changes of OWNER TO CURRENT/SESSION_USER
additions. Commit 31eae602 added new syntax to many DDL commands to
use CURRENT_USER or SESSION_USER instead of role name in ALTER ...
OWNER TO, but because of a misplaced '{', the syntax in the docs
implied that the syntax was "ALTER ... CURRENT_USER", instead of
"ALTER ... OWNER TO CURRENT_USER". Fix that, and also the funny
indentation in some of the modified syntax blurps.
http://git.postgresql.org/pg/commitdiff/cba045b0bd25285242936fd678bc443bfd0d5b83

Tom Lane pushed:

- Fix portability issue in pg_upgrade test script: avoid $PWD.
SUSv2-era shells don't set the PWD variable, though anything more
modern does. In the buildfarm environment this could lead to
test.sh executing with PWD pointing to $HOME or another high-level
directory, so that there were conflicts between concurrent
executions of the test in different branch subdirectories. This
appears to be the explanation for recent intermittent failures on
buildfarm members binturong and dingo (and might well have something
to do with the buildfarm script's failure to capture log files from
pg_upgrade tests, too). To fix, just use `pwd` in place of $PWD.
AFAICS test.sh is the only place in our source tree that depended on
$PWD. Back-patch to all versions containing this script. Per
buildfarm. Thanks to Oskari Saarenmaa for diagnosing the problem.
http://git.postgresql.org/pg/commitdiff/9a8f58331067e18a5dc10670e687f21ae6a2172e

- Fix postmaster's handling of a startup-process crash. Ordinarily, a
failure (unexpected exit status) of the startup subprocess should be
considered fatal, so the postmaster should just close up shop and
quit. However, if we sent the startup process a SIGQUIT or SIGKILL
signal, the failure is hardly "unexpected", and we should attempt
restart; this is necessary for recovery from ordinary backend
crashes in hot-standby scenarios. I attempted to implement the
latter rule with a two-line patch in commit
442231d7f71764b8c628044e7ce2225f9aa43b67, but it now emerges that
that patch was a few bricks shy of a load: it failed to distinguish
the case of a signaled startup process from the case where the new
startup process crashes before reaching database consistency. That
resulted in infinitely respawning a new startup process only to have
it crash again. To handle this properly, we really must track
whether we have sent the *current* startup process a kill signal.
Rather than add yet another ad-hoc boolean to the postmaster's
state, I chose to unify this with the existing RecoveryError flag
into an enum tracking the startup process's state. That seems more
consistent with the postmaster's general state machine design.
Back-patch to 9.0, like the previous patch.
http://git.postgresql.org/pg/commitdiff/45811be94e8539190b5e1a4f2cbdfef97fa391b5

- Improve documentation about array concat operator vs. underlying
functions. The documentation implied that there was seldom any
reason to use the array_append, array_prepend, and array_cat
functions directly. But that's not really true, because they can
help make it clear which case is meant, which the || operator can't
do since it's overloaded to represent all three cases. Add some
discussion and examples illustrating the potentially confusing
behavior that can ensue if the parser misinterprets what was meant.
Per a complaint from Michael Herold. Back-patch to 9.2, which is
where || started to behave this way.
http://git.postgresql.org/pg/commitdiff/e4f29ce32391525629c75aade86f2f939956c69c

- Add now-required #include. Fixes compiler warning induced by
808ea8fc7bb259ddd810353719cac66e85a608c8.
http://git.postgresql.org/pg/commitdiff/0a0fe2ff6ef65e3a1cf4d83d96eab144477a0220

- Fix assorted memory leaks. Per Coverity (not that any of these are
so non-obvious that they should not have been caught before commit).
The extent of leakage is probably minor to unnoticeable, but a leak
is a leak. Back-patch as necessary. Michael Paquier
http://git.postgresql.org/pg/commitdiff/bcc87b6b00de5b36984e0b43a78a8377a3577548

Joe Conway pushed:

- Improve regression test coverage of table lock modes vs permissions.
Test the interactions with permissions and LOCK TABLE. Specifically
ROW EXCLUSIVE, ACCESS SHARE, and ACCESS EXCLUSIVE modes against
SELECT, INSERT, UPDATE, DELETE, and TRUNCATE permissions. Discussed
by Stephen Frost and Michael Paquier, patch by the latter. Backpatch
to 9.5 where matching behavior was first committed.
http://git.postgresql.org/pg/commitdiff/e66a45344ff33d64aa6ff50673ff9fe8577ea6db

- Add assign_expr_collations() to CreatePolicy() and AlterPolicy().
As noted by Noah Misch, CreatePolicy() and AlterPolicy() omit to
call assign_expr_collations() on the node trees. Fix the omission
and add his test case to the rowsecurity regression test.
http://git.postgresql.org/pg/commitdiff/808ea8fc7bb259ddd810353719cac66e85a608c8

Noah Misch pushed:

- Given a gcc-compatible xlc compiler, prefer xlc-style atomics. This
evades a ppc64le "IBM XL C/C++ for Linux" compiler bug. Back-patch
to 9.5, where the atomics facility was introduced.
http://git.postgresql.org/pg/commitdiff/1e700e0fa02b3fd8990d4eaa8bd51b9352cc9736

- Replace use of "diff -q". POSIX does not specify the -q option, and
many implementations do not offer it. Don't bother changing the
MSVC build system, because having non-GNU diff on Windows is
vanishingly unlikely. Back-patch to 9.2, where this invocation was
introduced.
http://git.postgresql.org/pg/commitdiff/b28244abe58382be980d9b9577276c0700dd2cbc

- Fix null pointer dereference in "\c" psql command. The psql crash
happened when no current connection existed. (The second new check
is optional given today's undocumented NULL argument handling in
PQhost() etc.) Back-patch to 9.0 (all supported versions).
http://git.postgresql.org/pg/commitdiff/e135c3d99f4fcb03efa7b0b64445853ee2a74857

- Link pg_stat_statements with libm. The AIX 7.1 libm is static, and
AIX postgres executables do not export symbols acquired from
libraries. Back-patch to 9.5, where commit
cfe12763c32437bc708a64ce88a90c7544f16185 added a sqrt() call.
http://git.postgresql.org/pg/commitdiff/0689cfc34b09136b9daee9c60c85fbed21b62370

- Add .gitignore entries for AIX-specific intermediate build
artifacts.
http://git.postgresql.org/pg/commitdiff/bfb4cf12abc14a99b29c9c3f768b0c7f568c262d

- Finish generic-xlc.h draft atomics implementation. Back-patch to
9.5, where commit b64d92f1a5602c55ee8b27a7ac474f03b7aee340
introduced this file.
http://git.postgresql.org/pg/commitdiff/0d32d2e693ed69b3080be91f734c547f703b49ad

- Revoke support for strxfrm() that write past the specified array
length. This formalizes a decision implicit in commit
4ea51cdfe85ceef8afabceb03c446574daa0ac23 and adds clean detection of
affected systems. Vendor updates are available for each such known
bug. Back-patch to 9.5, where the aforementioned commit first
appeared.
http://git.postgresql.org/pg/commitdiff/be8b06c364db315258fb074e9bfbf5db04ab2c5f

Bruce Momjian pushed:

- doc: fix typo in CREATE POLICY manual page. Backpatch through 9.5
http://git.postgresql.org/pg/commitdiff/57057e2124730b4663a02391369f0ac979791e32

== Rejected Patches (for now) ==

No one was disappointed this week :-)

== Pending Patches ==

Pavel Stehule sent in another revision of a patch to allow raw output
from COPY in libpq.

Ashutosh Bapat sent in a patch to set up a notification channels for
background workers, which can among other things spawn other
background workers.

Julien Rouhaud sent in another revision of a patch to add an
auto_explain sample ratio parameter.

Gurjeet Singh sent in a patch to add SlotIsPhyscial/SlotIsLogical
macros and add a boolean for whether immediately to reserve an LSN for
CREATE_REPLICATION_SLOT ... PHYSICAL.

David Rowley sent in another revision of a patch to improve
performance for joins where outer side is unique.

Michael Paquier sent in another revision of a patch to add a TAP test
for pg_dump checking data dump of extension tables.

Pavel Stehule and Merlin Moncure traded patches to add a
client_min_messages_context GUC.

SAWADA Masahiko sent in four more revisions of a patch to a "frozen"
bit to the visibility map.

Marko Tiikkaja sent in two revisions of a patch to expose
confirmed_flush in pg_stat_replication_slots.

Pavel Stehule sent in a patch to allow tab completing boolean and enum
variables if no specific tab completion already exists.

Amit Langote sent in a patch to correct some comments in predtest.c.

Satoshi Nagayasu sent in a patch to expose the log_disconnections GUC
variable outside postgres.c.

Ildus Kurbangaliev sent in a patch to help monitor waits.

David Rowley sent in another revision of a patch to allow an
aggregate's state to be shared among other aggregate functions when
both aggregate's transition functions (and a few other things) match.

Ashutosh Bapat sent in a patch to create a transaction manager for
FDWs.

Michael Paquier sent in another revision of a patch to support TAP
tests with MSVC and Windows.

Stephen Frost sent in a patch to help secure copy.c handling for RLS.

Alexander Korotkov and Michael Paquier traded patches to add a
fillfactor for GIN indexes.

Kyotaro HORIGUCHI sent in a flock of patches implementing asynchronous
execution on FDWs.

Etsuro Fujita sent in a patch to modify create_foreignscan_plan so
that it detects whether any system columns are requested if scanning a
base relation.

Heikki Linnakangas sent in another revision of a patch to add some
functionality to ALTER OPERATOR.

Peter Geoghegan sent in a patch to reuse abbreviated keys during
second pass of ordered [set] aggregates.

Jeff Davis sent in another revision of a patch to implement memory
accounting for hash aggregates.

Tomas Vondra sent in a patch to enable index-only scans with partial
indexes.

Joe Conway sent in a patch to fix some infelicities in the interaction
between RLS and collations.

Pavel Stehule and Alexander Shulgin traded patches to add generalized
JSON output functions.

Michael Paquier sent in two more revisions of a patch to remove SSL
renegotiation.

Michael Paquier sent in a patch to fix some memory leaks.