== PostgreSQL Weekly News - May 18 2014 ==

== PostgreSQL Weekly News - May 18 2014 ==

PostgreSQL 9.4 Beta 1 released. TEST!
http://www.postgresql.org/developer/beta

PostgreSQL Conference Europe 2014 will be October 21-24 at Hotel
Miguel Angel in Madrid, Spain. The CfP is open for talks in English
and Spanish.
http://2014.pgconf.eu/

== PostgreSQL Product News ==

pg_catcheck, a catalog integrity checker for PostgreSQL and Postgres
Plus Advanced Server, released.
https://github.com/EnterpriseDB/pg_catcheck

PG Commander 1.3, a GUI made specifically for PostgreSQL on OSX, released.
https://eggerapps.at/pgcommander/

psqlODBC 09.03.0300 released.
http://www.postgresql.org/ftp/odbc/versions/

psycopg2 2.5.3, a Python connector for PostgreSQL, released.
http://initd.org/psycopg/articles/2014/05/13/psycopg-253-released/

== PostgreSQL Jobs for May ==

http://archives.postgresql.org/pgsql-jobs/2014-05/threads.php

== PostgreSQL Local ==

PGCon 2014, the world-wide developer conference for PostgreSQL, will
be in Ottawa, Ontario, Canada May 20-24, 2014.
http://www.pgcon.org/2014/

Char(14) and PGday UK will be held July 8 and 9, 2014.
http://www.char14.info
http://postgresqlusergroup.org.uk/

The sixth PGDay Cubano be held on 13 and 14 October 2014 in Habana.
https://postgresql.uci.cu/?p=380

Postgres Open 2014 will be in Chicago, IL, USA, September 17-19. The
CfP is open!
http://postgresopen.org/2014/callforpapers/

PostgreSQL Conference Europe 2014 will be held on October 21-24 in
Madrid, Spain, at the Hotel Miguel Angel.
http://2014.pgconf.eu/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david(at)fetter(dot)org, German language
to pwn(at)pgug(dot)de, Italian language to pwn(at)itpug(dot)org(dot) Spanish language
to pwn(at)arpug(dot)com(dot)ar(dot)

== Applied Patches ==

Heikki Linnakangas pushed:

- Free PQresult on error in pg_receivexlog. The leak is fairly small
and rare, but a leak nevertheless. Per Coverity report. Backpatch
to 9.2, where pg_receivexlog was added. pg_basebackup shares the
code, but it always exits on error, so there is no real leak.
http://git.postgresql.org/pg/commitdiff/c890b488063e23f12b5ab01c9a6d254a034fcea2

- Initialize padding bytes in btree_gist varbit support. The code
expands a varbit gist leaf key to a node key by copying the bit data
twice in a varlen datum, as both the lower and upper key. The lower
key was expanded to INTALIGN size, but the padding bytes were not
initialized. That's a problem because when the lower/upper keys are
compared, the padding bytes are used compared too, when the values
are otherwise equal. That could lead to incorrect query results.
REINDEX is advised for any btree_gist indexes on bit or bit varying
data type, to fix any garbage padding bytes on disk. Per Valgrind,
reported by Andres Freund. Backpatch to all supported versions.
http://git.postgresql.org/pg/commitdiff/540ac7cea919623f691b20892ccc50e5e33b5009

- Fix harmless access to uninitialized memory. When cache
invalidations arrive while ri_LoadConstraintInfo() is busy filling a
new cache entry, InvalidateConstraintCacheCallBack() compares the -
not yet initialized - oidHashValue field with the to-be-invalidated
hash value. To fix, check whether the entry is already marked as
invalid. Andres Freund
http://git.postgresql.org/pg/commitdiff/f35aef415aa755c4e99f8c0ef83f9d14dbc48bb4

- Silence warnings about redefining popen on Mingw-w64. Mingw-w64
headers map popen/pclose to _popen and _pclose, but we want to use
our popen wrapper rather than the Mingw-w64. #undef the Mingw's
version.
http://git.postgresql.org/pg/commitdiff/a82a17475d8a8a91df5fd1448b0f93772f590ecd

- Misc message style and doc fixes. Euler Taveira
http://git.postgresql.org/pg/commitdiff/ff810b4928882bfdc4ebe1ce603c47830aba3132

- Fix race condition in preparing a transaction for two-phase commit.
To lock a prepared transaction's shared memory entry, we used to
mark it with the XID of the backend. When the XID was no longer
active according to the proc array, the entry was implicitly
considered as not locked anymore. However, when preparing a
transaction, the backend's proc array entry was cleared before
transfering the locks (and some other state) to the prepared
transaction's dummy PGPROC entry, so there was a window where
another backend could finish the transaction before it was in fact
fully prepared. To fix, rewrite the locking mechanism of global
transaction entries. Instead of an XID, just have simple
locked-or-not flag in each entry (we store the locking backend's
backend id rather than a simple boolean, but that's just for
debugging purposes). The backend is responsible for explicitly
unlocking the entry, and to make sure that that happens, install a
callback to unlock it on abort or process exit. Backpatch to all
supported versions.
http://git.postgresql.org/pg/commitdiff/bb38fb0d43c8d7ff54072bfd8bd63154e536b384

- Handle duplicate XIDs in txid_snapshot. The proc array can contain
duplicate XIDs, when a transaction is just being prepared for
two-phase commit. To cope, remove any duplicates in
txid_current_snapshot(). Also ignore duplicates in the input
functions, so that if e.g. you have an old pg_dump file that already
contains duplicates, it will be accepted. Report and fix by Jan
Wieck. Backpatch to all supported versions.
http://git.postgresql.org/pg/commitdiff/8f9b9590d79fc1fc1ad08b207401acfdbb0bfac7

- Fix a couple of bugs in pg_recvlogical output to stdout. Don't
close stdout on SIGHUP. Also, when a SIGHUP is received, close the
file immediately, rather than only after receiving some more data
from the server. Rename a variable, to avoid mentally dealing with
double negatives (not unsynced means synced).
http://git.postgresql.org/pg/commitdiff/00c26b6a60e7ceed29ddae34b0a69fe945c08f81

- Initialize tsId and dbId fields in WAL record of COMMIT PREPARED.
Commit dd428c79 added dbId and tsId to the xl_xact_commit struct but
missed that prepared transaction commits reuse that struct. Fix
that. Because those fields were left unitialized, replaying a
commit prepared WAL record in a hot standby node would fail to
remove the relcache init file. That can lead to "could not open
file" errors on the standby. Relcache init file only needs to be
removed when a system table/index is rewritten in the transaction
using two phase commit, so that should be rare in practice. In HEAD,
the incorrect dbId/tsId values are also used for filtering in
logical replication code, causing the transaction to always be
filtered out. Analysis and fix by Andres Freund. Backpatch to 9.0
where hot standby was introduced.
http://git.postgresql.org/pg/commitdiff/07a4a93a0e35a778c77ffbbbc18de29e859e18f0

- Open output file before sleeping in pg_recvlogical. Let's complain
about e.g an invalid path or permission problem sooner rather than
later. Before this patch, we would only try to open the output file
after receiving the first decoded message from the server.
http://git.postgresql.org/pg/commitdiff/e7873b74d9c1bb7a325960f52043db6de88e6e66

- Fix thinko in logical decoding of commit-prepared records. The
decoding of prepared transaction commits accidentally used the XID
of the transaction performing the COMMIT PREPARED, not the XID of
the prepared transaction. Before bb38fb0d43c8d that lead to those
transactions not being decoded, afterwards to a assertion failure.
http://git.postgresql.org/pg/commitdiff/03e2b1017c0d360cae0b4d0a8bf5d6924536c834

- Fix test_decoding test case's check that slot has been dropped.
pg_stat_replication shows connected replication clients. The ddl
test case never has any replication clients connected, so querying
pg_stat_replication is pointless. To check that a slot has been
dropped correctly, query pg_replication_slots instead. Andres
Freund
http://git.postgresql.org/pg/commitdiff/afd0fcbc5d421454e7d55d949cca0721c9c135c3

- Add test case for logical decoding of prepared transactions. Andres
Freund
http://git.postgresql.org/pg/commitdiff/d900e192a3ef2dd73fae4522c35aadce6d72a534

- Update README, we don't do post-recovery cleanup actions anymore.
transam/README explained how B-tree incomplete splits were tracked
and fixed after recovery, as an example of handling complex actions
that need multiple WAL records, but that's not how it works anymore.
Explain the new paradigm.
http://git.postgresql.org/pg/commitdiff/a3655dd4a5cee3917a7d1766e07e36013e7e8835

Fujii Masao pushed:

- Fix typo in test_shm_mq document. Amit Langote
http://git.postgresql.org/pg/commitdiff/a3632771bb66eec74c7e73c0d7a5f8cdd6719be2

Tom Lane pushed:

- Ignore config.pl and buildenv.pl in src/tools/msvc. config.pl and
buildenv.pl can be used to customize build settings when using MSVC.
They should never get committed into the common source tree.
Back-patch to 9.0; it looks like the rules were different in 8.4.
Michael Paquier
http://git.postgresql.org/pg/commitdiff/73011f35eca7f4a3973289d5813648abbc1074a0

- Be more wary in choice of timezone names to test make_timestamptz
with. America/Metlakatla hasn't been in the IANA database all that
long, so some installations might not have it. It does seem
worthwhile to test with a fractional-minute GMT offset, but we can
get that from almost any pre-1900 date; I chose Europe/Paris, whose
LMT offset from Greenwich should be pretty darn well established.
Also, assuming that Mars/Mons_Olympus will never be in the IANA
database seems less than future-proof, so let's use a more fanciful
location for the bad-zone-name check. Per complaint from Christoph
Berg.
http://git.postgresql.org/pg/commitdiff/66b737cd9a72e10df4de1867ae7b675bc48d478c

- Make initdb throw error for bad locale values. Historically we've
printed a complaint for a bad locale setting, but then fallen back
to the environment default. Per discussion, this is not such a
great idea, because rectifying an erroneous locale choice
post-initdb (perhaps long after data has been loaded) could be
enormously expensive. Better to complain and give the user a chance
to double-check things. The behavior was particularly bad if the
bad setting came from environment variables rather than a bogus
command-line switch: in that case not only was there a fallback to
C/SQL_ASCII, but the printed complaint was quite unhelpful. It's
hard to be entirely sure what variables setlocale looked at, but we
can at least give a hint where the problem might be. Per a
complaint from Tomas Vondra.
http://git.postgresql.org/pg/commitdiff/31a263237fa3eb6f36d58335fbdb82bab3136db3

- Code review for recent changes in relcache.c. rd_replidindex should
be managed the same as rd_oidindex, and rd_keyattr and rd_idattr
should be managed like rd_indexattr. Omissions in this area meant
that the bitmapsets computed for rd_keyattr and rd_idattr would be
leaked during any relcache flush, resulting in a slow but permanent
leak in CacheMemoryContext. There was also a tiny probability of
relcache entry corruption if we ran out of memory at just the wrong
point in RelationGetIndexAttrBitmap. Otherwise, the fields were not
zeroed where expected, which would not bother the code any AFAICS
but could greatly confuse anyone examining the relcache entry while
debugging. Also, create an API function RelationGetReplicaIndex
rather than letting non-relcache code be intimate with the
mechanisms underlying caching of that value (we won't even mention
the memory leak there). Also, fix a relcache flush hazard
identified by Andres Freund: RelationGetIndexAttrBitmap must not
assume that rd_replidindex stays valid across index_open. The
aspects of this involving rd_keyattr date back to 9.3, so back-patch
those changes.
http://git.postgresql.org/pg/commitdiff/b23b0f5588d2e2f15edff66e072e339a8c9616a7

- Fix version check for pg_upgrade line type crosscheck. Per
buildfarm.
http://git.postgresql.org/pg/commitdiff/4456763c78d2a94c42e67ce788fbe94a07aa12dc

- In initdb, ensure stdout/stderr buffering behavior is what we
expect. Since this program may print to either stdout or stderr,
the relative ordering of its messages depends on the buffering
behavior of those files. Force stdout to be line-buffered and
stderr to be unbuffered, ensuring that the behavior will match
standard Unix interactive behavior, even when stdout and stderr are
rerouted to a file. Per complaint from Tomas Vondra. The
particular case he pointed out is new in HEAD, but issues of the
same sort could arise in any branch with other error messages, so
back-patch to all branches. I'm unsure whether we might not want to
do this in other client programs as well. For the moment, just fix
initdb.
http://git.postgresql.org/pg/commitdiff/2dc4f011fd61501cce507be78c39a2677690d44b

- Fix unportable setvbuf() usage in initdb. In yesterday's commit
2dc4f011fd61501cce507be78c39a2677690d44b, I tried to force buffering
of stdout/stderr in initdb to be what it is by default when the
program is run interactively on Unix (since that's how most manual
testing is done). This tripped over the fact that Windows doesn't
support _IOLBF mode. We dealt with that a long time ago in
syslogger.c by falling back to unbuffered mode on Windows. Export
that solution in port.h and use it in initdb. Back-patch to 8.4,
like the previous commit.
http://git.postgresql.org/pg/commitdiff/f62d417825d6a8e26768a130a5f4c9e454d9dbe4

- Fix valgrind warning for btree_gist indexes on macaddr. The macaddr
opclass stores two macaddr structs (each of size 6) in an index
column that's declared as being of type gbtreekey16, ie 16 bytes.
In the original coding this led to passing a palloc'd value of size
12 to the index insertion code, so that data would be fetched past
the end of the allocated value during index tuple construction.
This makes valgrind unhappy. In principle it could result in a
SIGSEGV, though with the current implementation of palloc there's no
risk since the 12-byte request size would be rounded up to 16 bytes
anyway. To fix, add a field to struct gbtree_ninfo showing the
declared size of the index datums, and use that in the palloc
requests; and use palloc0 to be sure that any wasted bytes are
cleanly initialized. Per report from Andres Freund. No back-patch
since there's no current risk of a real problem.
http://git.postgresql.org/pg/commitdiff/82bbb60c30dbff0633da34387ccab58d843379b5

- Fix a second cause of undersized pallocs for btree_gist indexes on
macaddr. gbt_macad_union also allocated 12-byte structs where we
really need 16. Per report from Andres Freund. No back-patch since
there's no current risk of a real problem.
http://git.postgresql.org/pg/commitdiff/39586bc1e971c136c4487b26eb7334dbbe23b102

- Suppress some more valgrind whining about btree_gist. A couple of
functions didn't bother to zero out pad bytes in datums that would
ultimately go to disk. Harmless, but valgrind doesn't know that.
http://git.postgresql.org/pg/commitdiff/af215d8190e6ab170c02c24afd1be81f5a147481

- Make sure chr(int) can't create invalid UTF8 sequences. Several
years ago we changed chr(int) so that if the database encoding is
UTF8, it would interpret its argument as a Unicode code point and
expand it into the appropriate multibyte sequence. However, we
weren't sufficiently careful about checking validity of the input.
According to RFC3629, UTF8 disallows code points above U+10FFFF
(note that the predecessor standard RFC2279 was more liberal).
Also, both versions of the UTF8 spec agree that Unicode
surrogate-pair codes should never appear in UTF8. Because our
encoding validity checks follow RFC3629, our failure to enforce
these restrictions in chr() means it could be used to produce text
strings that will be rejected when the database is dumped and
reloaded. To ensure consistency with the input functions, let's
actually apply pg_utf8_islegal() to the proposed output of chr().
Per discussion, this seems like too much of a behavioral change to
back-patch, but it's not too late to squeeze it into 9.4.
http://git.postgresql.org/pg/commitdiff/7894ac5004d5ec158f46296540d3cf49386d30e4

- Fix unaligned accesses in DecodeUpdate(). The xl_heap_header_len
structures in an XLOG_HEAP_UPDATE record aren't necessarily aligned
adequately. The regular replay function for these records is aware
of that, but decode.c didn't get the memo. I'm not sure why the
buildfarm failed to catch this; the test_decoding test certainly
blows up real good on my old HPPA box. Also, I'm pretty sure that
the address arithmetic was wrong for the case of
XLOG_HEAP_CONTAINS_OLD and not XLOG_HEAP_CONTAINS_NEW_TUPLE, though
this apparently can't happen when logical decoding is active.
http://git.postgresql.org/pg/commitdiff/6c42b2b10af3f717030966b9f05867f3e065becc

- Fix a bunch of functions that were declared static then defined
not-static. Per testing with a compiler that whines about this.
http://git.postgresql.org/pg/commitdiff/c1907f0cc49e38df9853b7547c9afce5204e4784

- Make fallback implementation of pg_memory_barrier() work. The
fallback implementation involves acquiring and releasing a spinlock
variable that is otherwise unreferenced --- not even to the extent
of initializing it. This accidentally fails to fail on platforms
where spinlocks should be initialized to zeroes, but elsewhere it
results in a "stuck spinlock" failure during startup. I griped
about this last July, and put in a hack that worked for gcc on HPPA,
but didn't get around to fixing the general case. Per the
discussion back then, the best thing to do seems to be to initialize
dummy_spinlock in main.c.
http://git.postgresql.org/pg/commitdiff/44cd47c1d49655c5dd9648bde8e267617c3735b4

- Fix two ancient memory-leak bugs in relcache.c.
RelationCacheInsert() ignored the possibility that
hash_search(HASH_ENTER) might find a hashtable entry already present
for the same OID. However, that can in fact occur during recursive
relcache load scenarios. When it did happen, we overwrote the
pointer to the pre-existing Relation, causing a session-lifespan
leakage of that entire structure. As far as is known, the
pre-existing Relation would always have reference count zero by the
time we arrive back at the outer insertion, so add code that deletes
the pre-existing Relation if so. If by some chance its refcount is
positive, elog a WARNING and allow the pre-existing Relation to be
leaked as before. Also, AttrDefaultFetch() was sloppy about leaking
the cstring form of the pg_attrdef.adbin value it's copying into the
relcache structure. This is only a query-lifespan leakage, and
normally not very significant, but it adds up during CLOBBER_CACHE
testing. These bugs are of very ancient vintage, but I'll refrain
from back-patching since there's no evidence that these leaks amount
to anything in ordinary usage.
http://git.postgresql.org/pg/commitdiff/078b2ed291c758e7125d72c3a235f128d40a232b

- Ooops, I broke initdb with that last patch. That's what I get for
not fully retesting the final version of the patch. The
replace_allowed cross-check needs an additional special case for
bootstrapping.
http://git.postgresql.org/pg/commitdiff/0c19aaba22c4d904f7259ef11f7d6db7b5dd36a0

Bruce Momjian pushed:

- doc: 9.4 release note adjustments. Report by Nicolas Barbier,
Tatsuo Ishii, MauMau
http://git.postgresql.org/pg/commitdiff/6de9a6c015d543c41933171cd6cf5d357597f19f

- docs: use structfield instead of structname in 9.4 release notes.
Where appropriate
http://git.postgresql.org/pg/commitdiff/9af8bd51a10edaf454c6b5246d79eded73528fd2

- docs: 9.4 release notes adjustments. Patch by Andres Freund,
slight adjustments by me
http://git.postgresql.org/pg/commitdiff/062f53518927f9bfe1820578ce79d3180b1aa2ca

- doc: auto-updatable view adjustments for 9.4 release notes. Report
by Dean Rasheed
http://git.postgresql.org/pg/commitdiff/e513087f6401a64b0d9f0b20cd2a51a7938ac645

- docs: mention windows quoting change in 9.4 release notes. Report
by Heikki Linnakangas
http://git.postgresql.org/pg/commitdiff/eb6144bb447ecbb036bfa08da85940cd50d49410

- doc: fix typo in 9.4 release note comments. Patch by Sergey
Muraviov
http://git.postgresql.org/pg/commitdiff/eefd90c4a95449f02b568d5370cfe6bcd1bf13e9

- doc: fix 9.4 release notes typo. Report by Dean Rasheed
http://git.postgresql.org/pg/commitdiff/ac53295d667e7727d7b70ddf11d82c067870501f

- pg_upgrade: error out on 'line' data type usage. The data type
internal format changed in 9.4. Also mention this in the 9.4
release notes.
http://git.postgresql.org/pg/commitdiff/bb797b640449d67f722d27660c718b68a692f8f9

- pg_update: slight major version adjustment
http://git.postgresql.org/pg/commitdiff/2e6602dfc25d1f8e8a2e1df48c96440b02aa19c1

Noah Misch pushed:

- Add Valgrind suppression for reorderbuffer padding bytes. Andres
Freund
http://git.postgresql.org/pg/commitdiff/5a90ac29c44f0dabf1e3a7b0443ba772cf206e2e

Peter Eisentraut pushed:

- pg_ctl: Write error messages to stderr
http://git.postgresql.org/pg/commitdiff/c424c049189f758920b1a3bb961bc05c93a900df

- doc: Clarify what files pg_basebackup omits from data directory
http://git.postgresql.org/pg/commitdiff/4773c70aa4c938c367ee179f7301f70a4efabc0e

- Fix whitespace
http://git.postgresql.org/pg/commitdiff/2f8e68bd7ea3523977567369056c4b3e67e44fdc

== Rejected Patches (for now) ==

No one was disappointed this week :-)

== Pending Patches ==

Kaigai Kouhei sent in two revisions of a patch to fix an error in the
regression tests for SEPgsql.

Thomas Munro and Craig Ringer traded patches to implement SKIP LOCKED
DATA.

Sergey Muraviov sent in three more revisions of a patch to fix psql's
wrapping in extended mode with the default pager.

Amit Kapila sent in two revisions of a patch to help scale shared
buffer eviction by reducing contention around BufFreelistLock.

Emre Hasegeli sent in another revision of a patch to add a selectivity
estimation for inet operators.

Jeff Janes sent in a patch to fix pg_dump warnings in the MinGW build.

Jeff Janes sent in a patch to fix an issue with autovacuum scheduling
starvation and frenzy.

Craig Ringer sent in a patch to fix an issue where dynamic background
workers with BGW_NEVER_RESTART worker restarted on FatalError.

Steve Singer and Andres Freund traded patches to fix an issue where
CREATE REPLICATION SLOT would fail on a timeout.

Andres Freund sent in two revisions of a patch to fix an issue where
%d in log_line_prefix doesn't work for bg/autovacuum workers.

Heikki Linnakangas sent in another revision of a patch to change the
WAL format with corresponding changes to the API.

Marko Kreen sent in a patch to clarify the crypto documentation for
SSL.

Heikki Linnakangas sent in a patch to fix a checksum error in recovery
with btree index.

David Rowley sent in another revision of a patch to allow join
removals for more join types.

Fabrízio de Royes Mello sent in a patch to fix an issue where psql's
\db+ does not have a size column.

Fabrízio de Royes Mello sent in a patch to fix a wrong name in the
documentation of the pg_stat_replication view.