== PostgreSQL Weekly News - February 15 2015 ==

== PostgreSQL Weekly News - February 15 2015 ==

== PostgreSQL Product News ==

pgpool-II versions 3.4.1, 3.3.5, 3.2.10, 3.1.13, and 3.0.17 released
with pgpoolAdmin 3.4.1.
pgpool-II is a connection pooler and replication system for PostgreSQL.
http://pgpool.net/mediawiki/index.php/Downloads
http://pgpool.net/mediawiki/index.php/Yum_Repository

psycopg2 2.6 and 2.5.5 Python connector for PostgreSQL, released.
http://initd.org/psycopg/articles/2015/02/09/psycopg-26-and-255-released/

== PostgreSQL Jobs for February ==

http://archives.postgresql.org/pgsql-jobs/2015-02/threads.php

== PostgreSQL Local ==

The Melbourne PostgreSQL meetup on February 18, 2015 will be hosting
Gabriele Bartolini on PostgreSQL 9.4 for devops. Details below, and
R, SVP.
http://www.meetup.com/melpug/events/219082475/

pgDaySF 2015 will be held March 10, 2015 in Burlingame, California.
http://sfpostgres.org/pgday-sf-2015-call-for-speakers-and-sponsors/

Registration for Nordic PostgreSQL Day, March 11, 2015 in Copenhagen,
Denmark, is open.
http://2015.nordicpgday.org/registration/

PGConf US 2015 takes place March 25-27, 2015 in NYC.
http://nyc.pgconf.us/2015/

PGCon 2015 is June 16-20 in Ottawa, Canada.
http://www.pgcon.org/2015/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david(at)fetter(dot)org, German language
to pwn(at)pgug(dot)de, Italian language to pwn(at)itpug(dot)org(dot) Spanish language
to pwn(at)arpug(dot)com(dot)ar(dot)

== Applied Patches ==

Fujii Masao pushed:

- Move pg_lzcompress.c to src/common. The meta data of PGLZ
symbolized by PGLZ_Header is removed, to make the compression and
decompression code independent on the backend-only varlena facility.
PGLZ_Header is being used to store some meta data related to the
data being compressed like the raw length of the uncompressed record
or some varlena-related data, making it unpluggable once PGLZ is
stored in src/common as it contains some backend-only code paths
with the management of varlena structures. The APIs of PGLZ are
reworked at the same time to do only compression and decompression
of buffers without the meta-data layer, simplifying its use for a
more general usage. On-disk format is preserved as well, so there
is no incompatibility with previous major versions of PostgreSQL for
TOAST entries. Exposing compression and decompression APIs of pglz
makes possible its use by extensions and contrib modules. Especially
this commit is required for upcoming WAL compression feature so that
the WAL reader facility can decompress the WAL data by using
pglz_decompress. Michael Paquier, reviewed by me.
http://git.postgresql.org/pg/commitdiff/40bede5477bb5bce98ce9548841cb414634c26f7

Heikki Linnakangas pushed:

- Fix MSVC build. When I moved pg_crc.c from src/port to src/common,
I forgot to modify MSVC build script accordingly.
http://git.postgresql.org/pg/commitdiff/cc761b170c5e7b4ef22ed918f4785ec1fabe62cd

- Move pg_crc.c to src/common, and remove pg_crc_tables.h. To get CRC
functionality in a client program, you now need to link with
libpgcommon instead of libpgport. The CRC code has nothing to do
with portability, so libpgcommon is a better home. (libpgcommon
didn't exist when pg_crc.c was originally moved to src/port.) Remove
the possibility to get CRC functionality by just #including
pg_crc_tables.h. I'm not aware of any extensions that actually did
that and couldn't simply link with libpgcommon. This also moves the
pg_crc.h header file from src/include/utils to src/include/common,
which will require changes to any external programs that currently
does #include "utils/pg_crc.h". That seems acceptable, as
include/common is clearly the right home for it now, and the change
needed to any such programs is trivial.
http://git.postgresql.org/pg/commitdiff/c619c2351f7ec429b6ddce519c939f7b8465d711

- Speed up CRC calculation using slicing-by-8 algorithm. This speeds
up WAL generation and replay. The new algorithm is significantly
faster with large inputs, like full-page images or when inserting
wide rows. It is slower with tiny inputs, i.e. less than 10 bytes or
so, but the speedup with longer inputs more than make up for that.
Even small WAL records at least have 24 byte header in the front.
The output is identical to the current byte-at-a-time computation,
so this does not affect compatibility. The new algorithm is only
used for the CRC-32C variant, not the legacy version used in tsquery
or the "traditional" CRC-32 used in hstore and ltree. Those are not
as performance critical, and are usually only applied over small
inputs, so it seems better to not carry around the extra lookup
tables to speed up those rare cases. Abhijit Menon-Sen
http://git.postgresql.org/pg/commitdiff/025c02420de990c15a90e9e3f86fcfbc5b59ee88

- Simplify waiting logic in reading from / writing to client. The
client socket is always in non-blocking mode, and if we actually
want blocking behaviour, we emulate it by sleeping and retrying. But
we have retry loops at different layers for reads and writes, which
was confusing. To simplify, remove all the sleeping and retrying
code from the lower levels, from be_tls_read and secure_raw_read and
secure_raw_write, and put all the logic in secure_read() and
secure_write().
http://git.postgresql.org/pg/commitdiff/80788a431e9bff06314a054109fdea66ac538199

- Simplify the way OpenSSL renegotiation is initiated in server. At
least in all modern versions of OpenSSL, it is enough to call
SSL_renegotiate() once, and then forget about it. Subsequent
SSL_write() and SSL_read() calls will finish the handshake. The
SSL_set_session_id_context() call is unnecessary too. We only have
one SSL context, and the SSL session was created with that to begin
with.
http://git.postgresql.org/pg/commitdiff/272923a0a6956187471df4f032eee06559520390

- Fix broken #ifdef for __sparcv8. Rob Rowan. Backpatch to all
supported versions, like the patch that added the broken #ifdef.
http://git.postgresql.org/pg/commitdiff/33e879c4e9f485405fc19b663392d8635714d020

Tom Lane pushed:

- Minor cleanup/code review for "indirect toast" stuff. Fix some
issues I noticed while fooling with an extension to allow an
additional kind of toast pointer. Much of this is just comment
improvement, but there are a couple of actual bugs, which might or
might not be reachable today depending on what can happen during
logical decoding. An example is that toast_flatten_tuple() failed
to cover the possibility of an indirection pointer in its input.
Back-patch to 9.4 just in case that is reachable now. In HEAD, also
correct some really minor issues with recent compression
reorganization, such as dangerously underparenthesized macros.
http://git.postgresql.org/pg/commitdiff/bc4de01db3a210e988dc88b585d7c38e6e7054c7

- Fix GEQO to not assume its join order heuristic always works. Back
in commit 400e2c934457bef4bc3cc9a3e49b6289bd761bc0 I rewrote GEQO's
gimme_tree function to improve its heuristic for modifying the given
tour into a legal join order. In what can only be called a fit of
hubris, I supposed that this new heuristic would *always* find a
legal join order, and ripped out the old logic that allowed
gimme_tree to sometimes fail. The folly of this is exposed by bug
#12760, in which the "greedy" clumping behavior of merge_clump() can
lead it into a dead end which could only be recovered from by
un-clumping. We have no code for that and wouldn't know exactly
what to do with it if we did. Rather than try to improve the
heuristic rules still further, let's just recognize that it *is* a
heuristic and probably must always have failure cases. So, put back
the code removed in the previous commit to allow for failure (but
comment it a bit better this time). It's possible that this code
was actually fully correct at the time and has only been broken by
the introduction of LATERAL. But having seen this example I no
longer have much faith in that proposition, so back-patch to all
supported branches.
http://git.postgresql.org/pg/commitdiff/1a179f36f7c612d3560a72396c559cd876dfab6b

- Fix pg_dump's heuristic for deciding which casts to dump. Back in
2003 we had a discussion about how to decide which casts to dump.
At the time pg_dump really only considered an object's containing
schema to decide what to dump (ie, dump whatever's not in
pg_catalog), and so we chose a complicated idea involving whether
the underlying types were to be dumped (cf commit
a6790ce85752b67ad994f55fdf1a450262ccc32e). But users are allowed to
create casts between built-in types, and we failed to dump such
casts. Let's get rid of that heuristic, which has accreted even
more ugliness since then, in favor of just looking at the cast's OID
to decide if it's a built-in cast or not. In passing, also fix some
really ancient code that supposed that it had to manufacture a
dependency for the cast on its cast function; that's only true when
dumping from a pre-7.3 server. This just resulted in some wasted
cycles and duplicate dependency-list entries with newer servers, but
we might as well improve it. Per gripes from a number of people,
most recently Greg Sabino Mullane. Back-patch to all supported
branches.
http://git.postgresql.org/pg/commitdiff/9feefedf9e92066fa6609d1e1e17b4892d81716f

- Fix more memory leaks in failure path in buildACLCommands. We
already had one go at this issue in commit d73b7f973db5ec7e, but we
failed to notice that buildACLCommands also leaked several
PQExpBuffers along with a simply malloc'd string. This time let's
try to make the fix a bit more future-proof by eliminating the
separate exit path. It's still not exactly critical because pg_dump
will curl up and die on failure; but since the amount of the
potential leak is now several KB, it seems worth back-patching as
far as 9.2 where the previous fix landed. Per Coverity, which
evidently is smarter than clang's static analyzer.
http://git.postgresql.org/pg/commitdiff/9179444d070713f53a97bc8f5c898a8304424c4d

- Fix minor memory leak in ident_inet(). We'd leak the ident_serv
data structure if the second pg_getaddrinfo_all (the one for the
local address) failed. This is not of great consequence because a
failure return here just leads directly to backend exit(), but if
this function is going to try to clean up after itself at all, it
should not have such holes in the logic. Try to fix it in a
future-proof way by having all the failure exits go through the same
cleanup path, rather than "optimizing" some of them. Per Coverity.
Back-patch to 9.2, which is as far back as this patch applies
cleanly.
http://git.postgresql.org/pg/commitdiff/58146d35deed3bd3e05ce4b5ba529e4e4a369ddb

- Fix missing PQclear() in libpqrcv_endstreaming(). This omission
leaked one PGresult per WAL streaming cycle, which possibly would
never be enough to notice in the real world, but it's still a leak.
Per Coverity. Back-patch to 9.3 where the error was introduced.
http://git.postgresql.org/pg/commitdiff/4f38a281a3d4a7cbcf9de2f676d44336d57b7717

- Avoid returning undefined bytes in chkpass_in(). We can't really
fix the problem that the result is defined to depend on random(), so
it is still going to fail the "unstable input conversion" test in
parse_type.c. However, we can at least satify valgrind. (It looks
like this code used to be valgrind-clean, actually, until somebody
did a careless s/strncpy/strlcpy/g on it.) In passing, let's just
make real sure that chkpass_out doesn't overrun its output buffer.
No need for backpatch, I think, since this is just to satisfy
debugging tools. Asif Naeem
http://git.postgresql.org/pg/commitdiff/80986e85aafd64165efc7f45b330f2f213713032

Michael Meskes pushed:

- Fixed array handling in ecpg. When ecpg was rewritten to the new
protocol version not all variable types were corrected. This patch
rewrites the code for these types to fix that. It also fixes the
documentation to correctly tell the status of array handling.
http://git.postgresql.org/pg/commitdiff/1f393fc923ec77f25fd37e16fd8ccb480df82ebb

Andres Freund pushed:

- Fix typo in logicaldecoding.sgml. Author: Tatsuo Ishii. Backpatch
to 9.4, where logical decoding was introduced.
http://git.postgresql.org/pg/commitdiff/8785e6e378f919827d851926ea0f400f08afd477

Bruce Momjian pushed:

- pg_upgrade: preserve freeze info for postgres/template1 dbs.
pg_database.datfrozenxid and pg_database.datminmxid were not
preserved for the 'postgres' and 'template1' databases. This could
cause missing clog file errors on access to user tables and indexes
after upgrades in these databases. Backpatch through 9.0
http://git.postgresql.org/pg/commitdiff/866f3017a88b9cb639f05c9ad413e5eafe0266e3

- pg_upgrade: improve checksum mismatch error message. Patch by Greg
Sabino Mullane, slight adjustments by me
http://git.postgresql.org/pg/commitdiff/dc01efa5ccdd9804b0c97d4d9b651ba9360ebb11

- pg_upgrade: quote directory names in delete_old_cluster script.
This allows the delete script to properly function when special
characters appear in directory paths, e.g. spaces. Backpatch
through 9.0
http://git.postgresql.org/pg/commitdiff/056764b10248bff702d9d7b8b97690668eaf1c93

Peter Eisentraut pushed:

- pg_regress: Write processed input/*.source into output dir. Before,
it was writing the processed files into the input directory, which
is incorrect in a vpath build.
http://git.postgresql.org/pg/commitdiff/64cdbbc48cade73d7b0831444a62e19fd4a342f8

Andrew Dunstan pushed:

- Minor housekeeping on JSON docs.
http://git.postgresql.org/pg/commitdiff/c5b31e29f0128b1c3c1caeeef81b0986cebdcd03

== Rejected Patches (for now) ==

No one was disappointed this week :-)

== Pending Patches ==

Tatsuo Ishii and Michael Paquier traded patches to fix some
incompatible options in pgbench.

David Steele sent in two more revisions of a patch to implement
pgaudit.

Jan Urbański sent in two revisions of a patch to fix broken
multi-threaded SSL callback handling in libpq.

Rahila Syed sent in three more revisions of a patch to implement
compression of full-page writes.

Peter Geoghegan and Heikki Linnakangas traded patches to implement
INSERT ... ON CONFLICT {UPDATE | IGNORE}.

Heikki Linnakangas sent in two revisions of a patch to use use Intel
SSE4.2 CRC instructions where available.

Michael Paquier sent in three more revisions of a patch to implement
table-level autovacuum_min_duration.

Michael Paquier sent in a patch to add a wal_retrieve_retry_interval.

Kyotaro HORIGUCHI sent in two revisions of a patch to fix an issue
where pg_basebackup fails silently.

Tom Lane sent in two revisions of a patch to manipulate complex types
as non-contiguous structures in-memory.

Etsuro Fujita sent in another revision of a patch to fix
ExplainModifyTarget to work as expected.

Michael Paquier sent in a patch to add palloc_extended for frontend
and backend, and another to rework handling of OOM when allocating
record buffer in XLOG reader.

Anastasia Lubennikova sent in two more revisions of a patch to allow
index-only scans for GiST indexes.

Emre Hasegeli sent in another revision of a patch to add a BRIN range
operator class.

Robert Haas sent in two more revisions of a patch to implement
parallel mode and parallel contexts.

Marco Nenciarini sent in another revision of a patch to implement
file-based incremental backup.

Marco Nenciarini sent in another revision of a patch to fix
pg_check_dir.

Michael Paquier sent in a patch to fix an odd assertion using
VACOPT_FREEZE in vacuum.c.

Teodor Sigaev sent in a patch to implement some of the intarray
features for all arrays.

Teodor Sigaev sent in another revision of a patch to add a compress
method for SP-GiST.

Heikki Linnakangas sent in another revision of a patch to redesign
checkpoint_segments.

Heikki Linnakangas sent in a patch to refactor unit conversions.

Atri Sharma sent in another revision of a patch to implement UPDATE
... SET(*) =.

Asif Naeem sent in a patch to fix chkpass to work under
RANDOMIZE_ALLOCATED_MEMORY.

Kevin Grittner sent in two revisions of a patch to fix an issue
manifesting as "multiple backends attempting to wait for pincount 1".

Kevin Grittner sent in a patch to help control bloat in the face of
old snapshots.

Peter Eisentraut sent in a patch to speed up make check-world.

Álvaro Herrera sent in another revision of a patch to add CREATE
ALTER, GRANT/REVOKE, COMMENT ON, and SECURITY LABEL command deparsing
support to event triggers.

Peter Eisentraut sent in a patch to add a pg_settings.pending_restart
column.

Peter Eisentraut sent in a patch to use forward slashes exclusively in
Windows builds, making it possible at least in theory to build such on
non-Windows machines.

Kevin Grittner sent in a patch to allow a "snapshot too old" error.

Andrew Dunstan sent in two revisions of a patch to add mogrify and
indent to JSONB.

Robert Haas sent in another revision of a patch to allow assessing
parallel safety.

Alexander Korotkov sent in another revision of a patch to implement
KNN-GiST with recheck.

Kaigai Kouhei sent in another revision of a patch to implement join
replacement with foreign-/custom-scan.

Petr (PJMODOS) Jelinek sent in another revision of a patch to
implement a sequence access method.

Tomas Vondra sent in a PoC patch to implement a mmap-based memory
context / allocator.

Emre Hasegeli sent in another revision of a patch to implement
selectivity estimation for inet operators.

Andres Freund sent in another revision of a patch to implement
replication identifiers.

David Steele sent in a patch to fix some dependency issues in the
documentation build for OSX.

Andrew (RhodiumToad) Gierth and Tomas Vondra traded patches to fix
some really bad blowups with hash outer join and nulls.