== PostgreSQL Weekly News - December 03 2017 ==

From: David Fetter <david(at)fetter(dot)org>
To: PostgreSQL Announce <pgsql-announce(at)postgresql(dot)org>
Subject: == PostgreSQL Weekly News - December 03 2017 ==
Date: 2017-12-03 19:31:08
Message-ID: 20171203193107.GB26891@fetter.org
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-announce

== PostgreSQL Weekly News - December 03 2017 ==

== PostgreSQL Product News ==

padnag 1.2.1, a system to synchronize PostgreSQL roles with Active Directory
users and groups, released.

== PostgreSQL Jobs for December ==


== PostgreSQL Local ==

PGConf Local: Austin will be held December 4 - 5, 2017. Program available
and registration is now open at

PGConf.ASIA 2017 will take place on December 4-6 2017 in Akihabara, Tokyo,

FOSDEM PGDay 2018, a one day conference held before the main FOSDEM event will
be held in Brussels, Belgium, on Feb 2nd, 2018.

Prague PostgreSQL Developer Day 2018 (P2D2 2018) is a two-day
conference that will be held on February 14-15 2018 in Prague, Czech Republic.
The CfP is open until January 5, 2018 at https://p2d2.cz/callforpapers

PGConf India 2018 will be on February 22-23, 2018 in Bengaluru, Karnataka.

PostgreSQL(at)SCaLE is a two day, two track event which takes place on
March 8-9, 2018, at Pasadena Convention Center, as part of SCaLE 16X.

Nordic PGDay 2018 will be held in Oslo, Norway, at the Radisson Blu Hotel
Nydalen, on March 13, 2018. The CfP is open through December 31, 2017 at

pgDay Paris 2018 will be held in Paris, France at the Espace Saint-Martin, on
March 15 2018. The CfP is open until December 31, 2017.

PGConf APAC 2018 will be held in Singapore March 22-24, 2018. The CfP is open
at http://2018.pgconfapac.org/cfp through December 4, 2017.

The German-speaking PostgreSQL Conference 2018 will take place on April 13th,
2018 in Berlin. The CfP is open until January 09, 2018 at
http://2018.pgconf.de/de/callforpapers.html and the conference site is at

PGCon 2018 will take place in Ottawa on May 29 - June 2018. The CfP goes out

PGConf.Brazil 2018 will take place in São Paulo, Brazil on August 3-4 2018. The
CfP will open soon.

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm EST5EDT. Please send English
language ones to david(at)fetter(dot)org, German language to pwn(at)pgug(dot)de, Italian
language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Magnus Hagander pushed:

- Fix typo in comment. Andreas Karlsson

Simon Riggs pushed:

- Pad XLogReaderState's per-buffer data_bufsz more aggressively. Originally, we
palloc'd this buffer just barely big enough to hold the largest xlog backup
block seen so far. We now MAXALIGN the palloc size. The original coding could
result in many repeated palloc cycles, in the worst case where we see a series
ofgradually larger xlog records. We ameliorate that similarly to
8735978e7aebfbc499843630131c18d1f7346c79 by imposing a minimum buffer size of
BLCKSZ. Discussion:

- Additional docs for toast_tuple_target changes.

Tom Lane pushed:

- Fix creation of resjunk tlist entries for inherited mixed UPDATE/DELETE.
rewriteTargetListUD's processing is dependent on the relkind of the query's
target table. That was fine at the time it was made to act that way, even for
queries on inheritance trees, because all tables in an inheritance tree would
necessarily be plain tables. However, the 9.5 feature addition allowing some
members of an inheritance tree to be foreign tables broke the assumption that
rewriteTargetListUD's output tlist could be applied to all child tables with
nothing more than column-number mapping. This led to visible failures if
foreign child tables had row-level triggers, and would also break in cases
where child tables belonged to FDWs that used methods other than CTID for row
identification. To fix, delay running rewriteTargetListUD until after the
planner has expanded inheritance, so that it is applied separately to the
(already mapped) tlist for each child table. We can conveniently call it from
preprocess_targetlist. Refactor associated code slightly to avoid the need to
heap_open the target relation multiple times during preprocess_targetlist.
(The APIs remain a bit ugly, particularly around the point of which steps
scribble on parse->targetList and which don't. But avoiding such scribbling
would require a change in FDW callback APIs, which is more pain than it's
worth.) Also fix ExecModifyTable to ensure that "tupleid" is reset to NULL
when we transition from rows providing a CTID to rows that don't. (That's
really an independent bug, but it manifests in much the same cases.) Add a
regression test checking one manifestation of this problem, which was that
row-level triggers on a foreign child table did not work right. Back-patch to
9.5 where the problem was introduced. Etsuro Fujita, reviewed by Ildus
Kurbangaliev and Ashutosh Bapat Discussion:

- Fix assorted syscache lookup sloppiness in partition-related code.
heap_drop_with_catalog and ATExecDetachPartition neglected to check for
SearchSysCache failures, as noted in bugs #14927 and #14928 from Pan Bian.
Such failures are pretty unlikely, since we should already have some sort of
lock on the rel at these points, but it's neither a good idea nor per project
style to omit a check for failure. Also, StorePartitionKey contained a
syscache lookup that it never did anything with, including never releasing the
result. Presumably the reason why we don't see refcount-leak complaints is
that the lookup always fails; but in any case it's pretty useless, so remove
it. All of these errors were evidently introduced by the relation
partitioning feature. Back-patch to v10 where that came in. Amit Langote and
Tom Lane Discussion:

- Mark some more functions as pg_attribute_noreturn(). Doing this suppresses
Coverity warnings and might allow improved code in some cases. The prospects
of that are not so bright as to warrant back-patching, though. Michael
Paquier, per Coverity

- Clarify old comment about qual_is_pushdown_safe's handling of subplans. This
comment glossed over the difference between initplans and subplans, but they
are indeed different for our purposes here.

- Fix neqjoinsel's behavior for semi/anti join cases. Previously, this function
estimated the selectivity as 1 minus eqjoinsel() for the negator equality
operator, regardless of join type (I think there was an expectation that
eqjoinsel would handle the join type). But actually this is completely wrong
for semijoin cases: the fraction of the LHS that has a non-matching row is not
one minus the fraction of the LHS that has a matching row. In reality a
semijoin with <> will nearly always succeed: it can only fail when the RHS is
empty, or it contains a single distinct value that is equal to the particular
LHS value, or the LHS value is null. The only one of those things we should
have much confidence in estimating is the fraction of LHS values that are
null, so let's just take the selectivity as 1 minus outer nullfrac. Per
coding convention, antijoin should be estimated the same as semijoin.
Arguably this is a bug fix, but in view of the lack of field complaints and
the risk of destabilizing plans, no back-patch. Thomas Munro, reviewed by
Ashutosh Bapat Discussion:

- Fix uninitialized-variable compiler warning induced by commit e4128ee76. I'm
a little bit astonished that anyone's compiler would have failed to complain
about this. The compiler surely does not know that is_procedure means the
function return value will be ignored.

Robert Haas pushed:

- Fix uninitialized-variable compiler warning induced by commit e4128ee76. I'm
a little bit astonished that anyone's compiler would have failed to complain
about this. The compiler surely does not know that is_procedure means the
function return value will be ignored.

- Add null test to partition constraint for default range partitions.
Non-default range partitions have a constraint which include null tests, and
both default and non-default list partitions also have a constraint which
includes null tests, but for some reason this was missed for default range
partitions. This could cause the partition constraint to evaluate to false
for rows that were (correctly) routed to that partition by insert tuple
routing, which could in turn cause constraint exclusion to prune the default
partition in cases where it should not. Amit Langote, reviewed by Kyotaro
Horiguchi Discussion:

- Teach bitmap heap scan to cope with absence of a DSA. If we have a plan that
uses parallelism but are unable to execute it using parallelism, for example
due to a lack of available DSM segments, then the EState's es_query_dsa will
be NULL. Parallel bitmap heap scan needs to fall back to a non-parallel scan
in such cases. Patch by me, reviewed by Dilip Kumar Discussion:

- Fix ReinitializeParallelDSM to tolerate finding no error queues. Commit
d4663350646ca0c069a36d906155a0f7e3372eb7 changed things so that shm_toc_lookup
would fail with an error rather than silently returning NULL in the hope that
such failures would be reported in a useful way rather than via a system
crash. However, it overlooked the fact that the lookup of
PARALLEL_KEY_ERROR_QUEUE in ReinitializeParallelDSM is expected to fail when
no DSM segment was created in the first place; in that case, we end up with a
backend-private memory segment that still contains an entry for
PARALLEL_KEY_FIXED but no others. Consequently a benign failure to initialize
parallelism can escalate into an elog(ERROR); repair. Discussion:

- If a range-partitioned table has no default partition, reject null keys.
Commit 4e5fe9ad19e14af360de7970caa8b150436c9dec introduced this problem. Also
add a test so it doesn't get broken again. Report by Rushabh Lathia. Fix by
Amit Langote. Reviewed by Rushabh Lathia and Amul Sul. Tweaked by me.

- Fix wrong function name in comment. Rushabh Lathia Discussion:

- Update typedefs.list and re-run pgindent. Discussion:

- Add extensive tests for partition pruning. Currently, partition pruning
happens via constraint exclusion, but there are pending places to replace that
with a different and hopefully faster mechanism. To be sure that we don't
change behavior without realizing it, add extensive test coverage. Note that
not all of these behaviors are optimal; in some cases, partitions are not
pruned even though it would be safe to do so. These tests therefore serve to
memorialize the current state rather than the ideal state. Patches that
improve things can update the test results as appropriate. Amit Langote,
adjusted by me. Review and testing of the larger patch set of which this is a
part by Ashutosh Bapat, David Rowley, Dilip Kumar, Jesper Pedersen, Rajkumar
Raghuwanshi, Beena Emerson, Amul Sul, and Kyotaro Horiguchi. Discussion:

- Make create_unique_path manage memory like mark_dummy_rel. Put the unique
path in the same context as the owning RelOptInfo, rather than the toplevel
planner context. This is how this function worked originally, but commit
f41803bb39bc2949db200116a609fd242d0ec221 changed it without explanation.
mark_dummy_rel adopted the older (or newer?) technique in commit
eca75a12a27d28b972fc269c1c8813cd8eb15441, which also featured a much better
explanation of why it is correct. So, switch back to that technique here,
with the same explanation given there. Although this fixes a possible memory
leak when GEQO is in use, the leak is minor and probably nobody cares, so no
back-patch. Ashutosh Bapat, reviewed by Tom Lane and by me Discussion:

- Remove extra word from comment. David Rowley, who also was the primary author
of the patch that added this function; the attribution in my previous commit,
84940644de931f331433b35e3a391822671f8c9c, was incorrect due to sloppiness on
my part. Discussion:

- New C function: bms_add_range. This will be used by pending patches to
improve partition pruning. Amit Langote and Kyotaro Horiguchi, per a
suggestion from David Rowley. Review and testing of the larger patch set of
which this is a part by Ashutosh Bapat, David Rowley, Dilip Kumar, Jesper
Pedersen, Rajkumar Raghuwanshi, Beena Emerson, Amul Sul, and Kyotaro
Horiguchi. Discussion:

- Fix uninitialized memory reference. Without this, when partdesc->nparts == 0,
we end up calling ExecBuildSlotPartitionKeyDescription without initializing
values and isnull. Reported by Coverity via Michael Paquier. Patch by
Michael Paquier, reviewed and revised by Amit Langote. Discussion:

- Try to exclude partitioned tables in toto. Ashutosh Bapat, reviewed by Jeevan
Chalke. Comment by me. Discussion:

- Re-allow INSERT .. ON CONFLICT DO NOTHING on partitioned tables. Commit
8355a011a0124bdf7ccbada206a967d427039553 was reverted in
f05230752d53c4aa74cffa9b699983bbb6bcb118, but this attempt is hopefully
better-considered: we now pass the correct value to ExecOpenIndices, which
should avoid the crash that we hit before. Amit Langote, reviewed by Simon
Riggs and by me. Some final editing by me. Discussion:

- postgres_fdw: Fix test that didn't test what it claimed. Antonin Houska
reported that the planner does consider pushing postgres_fdw_abs() to the
remote side, which happens because we make it shippable earlier in the test
case file. Jeevan Chalke provided this patch, which changes the join
condition to use random(), which is not shippable, instead. Antonin reviewed
the patch. Discussion: http://postgr.es/m/15265.1511985971@localhost

- Minor code beautification in partition_bounds_equal. Use get_greatest_modulus
more consistently, instead of doing the same thing in an ad-hoc manner in this
one place. Ashutosh Bapat Discussion:

Peter Eisentraut pushed:

- PL/Python: Fix potential NULL pointer dereference. After
d0aa965c0a0ac2ff7906ae1b1dad50a7952efa56, one error path in
PLy_spi_execute_fetch_result() could result in the variable "result" being
dereferenced after being set to NULL. To fix that, just clear the resources
right there and return early. Also add another SPI_freetuptable() call so
that that is cleared in all error paths. discovered by John Naylor
<jcnaylor(at)gmail(dot)com> via scan-build

- Revert "PL/Python: Fix potential NULL pointer dereference". This reverts
commit e42e2f38907681c48c43f49c5ec9f9f41a9aa9a5. It's not safe to return in
the middle of a PG_TRY block, so this will have to be done differently.

- Add compiler hints to PLy_elog(). Decorate PLy_elog() in a similar way as
elog(), to give compilers and static analyzers hints in which cases it does
not return. Reviewed-by: John Naylor <jcnaylor(at)gmail(dot)com>

- PL/Python: Fix remaining scan-build warnings. Apparently, scan-build thinks
that proc->is_setof can change during PLy_exec_function(). To make it
clearer, save the value in a local variable. Also add an assertion to clear
another warning. Reviewed-by: John Naylor <jcnaylor(at)gmail(dot)com>

- SQL procedures. This adds a new object type "procedure" that is similar to a
function but does not have a return type and is invoked by the new CALL
statement instead of SELECT or similar. This implementation is aligned with
the SQL standard and compatible with or similar to other SQL implementations.
This commit adds new commands CALL, CREATE/ALTER/DROP PROCEDURE, as well as
ALTER/DROP ROUTINE that can refer to either a function or a procedure (or an
aggregate function, as an extension to SQL). There is also support for
procedures in various utility commands such as COMMENT and GRANT, as well as
support in pg_dump and psql. Support for defining procedures is available in
all the languages supplied by the core distribution. While this commit is
mainly syntax sugar around existing functionality, future features will rely
on having procedures as a separate object type. Reviewed-by: Andrew Dunstan

- pg_basebackup: Fix progress messages when writing to a file. The progress
messages print out \r to keep overwriting the same line on the screen. But
this does not yield useful results when writing the output to a file. So in
that case, print out \n instead. Author: Martín Marqués
<martin(at)2ndquadrant(dot)com> Reviewed-by: Arthur Zakirov

- Check channel binding flag at end of SCRAM exchange. We need to check whether
the channel-binding flag encoded in the client-final-message is the same one
sent in the client-first-message. Reviewed-by: Michael Paquier

Álvaro Herrera pushed:

- Fix extstat collection when no stats are produced for a column. This is a
mistakenly placed conditional in bf2a691e02d7. Reported by Justin Pryzby
Discussion: https://postgr.es/m/20171117214352.GE25796@telsasoft.com

- Make memset() use sizeof() rather than re-compute size. This is simpler and
more closely follows overwhelming precedent. Report and patch by Mark Dilger.
Discussion: https://postgr.es/m/9A68FB88-5F45-4848-9926-8586E2D777D1@gmail.com

Andres Freund pushed:

- Add a barrier primitive for synchronizing backends. Provide support for
dynamic or static parties of processes to wait for all processes to reach
point in the code before continuing. This is similar to the mechanism of the
same name in POSIX threads and MPI, though has explicit phasing and dynamic
party support like the Java core library's Phaser. This will be used by an
upcoming patch adding support for parallel hash joins. Author: Thomas Munro
Reviewed-By: Andres Freund Discussion:

- Add some regression tests that exercise hash join code. Although hash joins
are already tested by many queries, these tests systematically cover the four
different states we can reach as part of the strategy for respecting work_mem.
Author: Thomas Munro Reviewed-By: Andres Freund

- Add infrastructure for sharing temporary files between backends.
SharedFileSet allows temporary files to be created by one backend and then
exported for read-only access by other backends, with clean-up managed by
reference counting associated with a DSM segment. This includes changes to
fd.c and buffile.c to support the new kind of temporary file. This will be
used by an upcoming patch adding support for parallel hash joins. Author:
Thomas Munro Reviewed-By: Peter Geoghegan, Andres Freund, Robert Haas, Rushabh
Lathia Discussion:

- Adjust #ifdef EXEC_BACKEND RemovePgTempFilesInDir() call. Other callers were
adjusted in the course of dc6c4c9dc2a111519b76b22daaaac86c5608223b. Per

Noah Misch pushed:

- Fix non-GNU makefiles for AIX make. Invoking the Makefile without an explicit
target was building every possible target instead of just the "all" target.
Back-patch to 9.3 (all supported versions).

== Pending Patches ==

Andres Freund sent in a patch to implement expression based aggregate transition
/ combine function invocation, part of the JIT infrastructure.

Oliver Ford, Erik Rijkers, and David Fetter traded patches to add RANGE with
values and exclusions clauses to the window functions.

Andreas Karlsson sent in another revision of a patch to add support for GnuTLS.

Alexander Korotkov sent in two revisions of a patch to implement atomic pgrename
on Windows.

Tomas Vondra sent in another revision of a patch to implement multivariate
histograms and MCV lists.

Jing Wang sent in another revision of a patch to implement COMMENT ON DATABASE

Daniel Gustafsson sent in two more revisions of a patch to refactor identifier
checks to consistently use strcmp.

Masahiko Sawada sent in three more revisions of a patch to move relation
extension locks out of the heavyweight lock manager.

Anthony Bykov sent in another revision of a patch to implement transforms for
JSON for PL/Perl.

Thomas Munro sent in another revision of a patch to implement parallel hash.

Ildus Kurbangaliev sent in another revision of a patch to implement custom
compression methods.

Yura Sokolov sent in a patch to create a header for customized qsort.

Feike Steenbergen sent in two revisions of a patch to skip index cleanup if
autovacuum did not do any work.

Doug Rady sent in another revision of a patch to pgbench to add an option to
build using ppoll() for larger connection counts.

Victor Drobny sent in two more revisions of a patch to add a new function for
tsquery creation.

Amul Sul sent in another revision of a patch to implement parallel append.

Sergei Kornilov sent in two revisions of a patch to use an index or check if

Amit Kapila sent in two revisions of a patch to fix the instrumentation for
parallel workers.

Thomas Munro sent in a patch to ensure that TupleDescCopy clears atthasdef,
attnotnull, and attidentity.

Andres Freund sent in a patch to implement expression-based grouping equality,
part of the infrastructure for JIT.

Amul Sul sent in another revision of a patch to restrict concurrent
update/delete with UPDATE of partition key.

Alexander Korotkov sent in another revision of a patch to fix contrib/cube's

Petr Jelínek sent in a patch to fix walsender timeouts when decoding large

Shubham Barai sent in another revision of a patch to implement predicate locking
in GiST indexes.

Nikolay Shaplov sent in another revision of a patch to move all am-related
reloption code into src/backend/access/[am-name] and get rid of relopt_kind for
custom AM.

Emre Hasegeli sent in another revision of a patch to improve the geometric

Amit Khandekar sent in another revision of a patch to see to it that UPDATEs of
a partition key which would cause a tuple to move to another partition do so.

Stas Kelvich and Petr Jelínek traded patches to make XactLockTableWait work for
transactions that are not yet self locked.

Beena Emerson sent in two more revisions of a patch to implement runtime
partition pruning.

David Rowley sent in another revision of a patch to remove [Merge]Append nodes
which contain a single subpath.

Andrey Borodin sent in a patch to update the patch for covering + unique indexes
to account for the new amcheck.

Pavel Stěhule sent in another revision of a patch to add extra checks to

David Rowley sent in another revision of a patch to see to it that LEFT JOIN is
removed in appropriate cases where DISTINCT also appears.

Alexey Chernyshov sent in a patch to make ASCII NUL a valid character.

Robert Haas sent in another revision of a patch to remove the IndexTupleDSize

Alexander Korotkov sent in another revision of a patch to implement incremental

Aleksey Kondratov sent in another revision of a patch to allow COPY to handle
certain types of errors more gracefully.

Michaël Paquier and Amit Langote traded patches to fix a use of uninitialized
variables in ExecFindPartition() for a parent partition without leaves.

Michaël Paquier sent in two more revisions of a patch to allow SSL connection
from a v11 client to v10 server with SCRAM channel binding.

Michaël Paquier sent in another revision of a patch to move SCRAM-related name
definitions to scram-common.h, add a "scramchannelbinding" connection parameter,
and implement channel binding tls-server-end-point for SCRAM.

Fabien COELHO sent in another revision of a patch to add more functions and
operators to pgbench.

Robert Haas sent in another revision of a patch to fix the superuser checks in
the PostgreSQL FDW.

Rajkumar Raghuwanshi sent in another revision of a patch to implement
partition-wise join for join between (declaratively) partitioned tables.

Peter Eisentraut sent in another revision of a patch to implement transaction
control in PROCEDUREs.

Nathan Bossart sent in two more revisions of a patch to add logging for VACUUM

Thomas Munro and Justin Pryzby traded patches to clarify the documentation on
huge pages.

Raúl Marín Rodríguez sent in another revision of a patch to add pow() support
for pgbench.

Chen Huajun sent in another revision of a patch to make pg_rewind to not copy
useless WAL files.

Ashutosh Bapat sent in another revision of a patch to implement an advanced
partition matching algorithm for partition-wise join.

Browse pgsql-announce by date

  From Date Subject
Next Message Nicolas Thauvin 2017-12-04 15:20:10 pitrery 2.0
Previous Message David Fetter 2017-11-26 21:16:00 == PostgreSQL Weekly News - November 26 2017 ==