== PostgreSQL Weekly News - April 01 2018 ==

== PostgreSQL Weekly News - April 01 2018 ==

A meta-early-April news item is replacing the usual early-April shenanigans in
today's newsletter.

== PostgreSQL Product News ==

ora_migrator, a tool to ease migration from Oracle to PostgreSQL, released.
https://github.com/cybertec-postgresql/ora_migrator

pgBackRest 2.01, a backup and restore system for PostgreSQL, released.
https://pgbackrest.org/release.html#2.01

== PostgreSQL Jobs for April ==

http://archives.postgresql.org/pgsql-jobs/2018-04/

== PostgreSQL Local ==

The German-speaking PostgreSQL Conference 2018 will take place on April 13th,
2018 in Berlin.
http://2018.pgconf.de/

PGConfNepal 2018 will be held May 4-5, 2018 at Kathmandu University, Dhulikhel,
Nepal.
https://postgresconf.org/conferences/Nepal2018

PGCon 2018 will take place in Ottawa on May 29 - June 1, 2018.
https://www.pgcon.org/2018/

Swiss PGDay 2018 will take place in Rapperswil (near Zurich) on June 29, 2018.
The CfP is open through April 14, 2018, and registration is open through June
28, 2018.
http://www.pgday.ch/2018/

PGConf.Brazil 2018 will take place in São Paulo, Brazil on August 3-4 2018.
http://pgconf.com.br

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm EST5EDT. Please send English
language ones to david(at)fetter(dot)org, German language to pwn(at)pgug(dot)de, Italian
language to pwn(at)itpug(dot)org(dot)

== Applied Patches ==

Peter Eisentraut pushed:

- initdb: Further polishing of --wal-segsize option. Extend documentation.
Improve option parsing in case no argument was specified.
https://git.postgresql.org/pg/commitdiff/8ad8d916f99d19e0be7800992c828c3c1a01b693

- pg_resetwal: Allow users to change the WAL segment size. This adds a new
option --wal-segsize (analogous to initdb) that changes the WAL segment size
in pg_control. Author: Nathan Bossart <bossartn(at)amazon(dot)com>
https://git.postgresql.org/pg/commitdiff/bf4a8676c316c177f395b54d3305ea4ccc838a66

- libpq: PQhost to return active connected host or hostaddr. Previously, PQhost
didn't return the connected host details when the connection type was
CHT_HOST_ADDRESS (i.e., via hostaddr). Instead, it returned the complete host
connection parameter (which could contain multiple hosts) or the default host
details, which was confusing and arguably incorrect. Change this to return
the actually connected host or hostaddr irrespective of the connection type.
When hostaddr but no host was specified, hostaddr is now returned. Never
return the original host connection parameter, and document that PQhost cannot
be relied on before the connection is established. PQport is similarly
changed to always return the active connection port and never the original
connection parameter. Author: Hari Babu <kommi(dot)haribabu(at)gmail(dot)com>
Reviewed-by: Michael Paquier <michael(at)paquier(dot)xyz> Reviewed-by: Kyotaro
HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> Reviewed-by: David G. Johnston
<david(dot)g(dot)johnston(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/1944cdc98273dbb8439ad9b387ca2858531afcf0

- Fix jsonb_plpython tests on older Python versions. Rewrite one test to avoid
a case where some Python versions have output format differences (Decimal('1')
vs Decimal("1")).
https://git.postgresql.org/pg/commitdiff/e81fc9b9dbf9d744dcc9fb210e4353a350be1e22

- Attempt to fix jsonb_plpython build on Windows.
https://git.postgresql.org/pg/commitdiff/75e95dd79ba22e18687a069d2ff2fd29afab5798

- PL/pgSQL: Nested CALL with transactions. So far, a nested CALL or DO in
PL/pgSQL would not establish a context where transaction control statements
were allowed. This fixes that by handling CALL and DO specially in PL/pgSQL,
passing the atomic/nonatomic execution context through and doing the required
management around transaction boundaries. Reviewed-by: Tomas Vondra
<tomas(dot)vondra(at)2ndquadrant(dot)com>
https://git.postgresql.org/pg/commitdiff/d92bc83c48bdea9888e64cf1e2edbac9693099c9

- Allow committing inside cursor loop. Previously, committing or aborting
inside a cursor loop was prohibited because that would close and remove the
cursor. To allow that, automatically convert such cursors to holdable cursors
so they survive commits or rollbacks. Portals now have a new state
"auto-held", which means they have been converted automatically from pinned.
An auto-held portal is kept on transaction commit or rollback, but is still
removed when returning to the main loop on error. This supports all languages
that have cursor loop constructs: PL/pgSQL, PL/Python, PL/Perl. Reviewed-by:
Ildus Kurbangaliev <i(dot)kurbangaliev(at)postgrespro(dot)ru>
https://git.postgresql.org/pg/commitdiff/056a5a3f63f1a29d9266165ee6e25c6a51ddd63c

- Transforms for jsonb to PL/Python. Add a new contrib module jsonb_plpython
that provide a transform between jsonb and PL/Python. jsonb values are
converted to appropriate Python types such as dicts and lists, and vice versa.
Author: Anthony Bykov <a(dot)bykov(at)postgrespro(dot)ru> Reviewed-by: Aleksander
Alekseev <a(dot)alekseev(at)postgrespro(dot)ru> Reviewed-by: Nikita Glukhov
<n(dot)gluhov(at)postgrespro(dot)ru>
https://git.postgresql.org/pg/commitdiff/3f44e3db72ad4097aae078c075a9b3cb3d6b761b

Tom Lane pushed:

- Fix unsafe extraction of the OID part of a relation filename. Commit
8694cc96b did this randomly differently from other callers of
parse_filename_for_nontemp_relation(). Perhaps unsurprisingly, the randomly
different way is wrong; it fails to ensure the extracted string is
null-terminated. Per buildfarm member skink. Discussion:
https://postgr.es/m/14453.1522001792@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/d0c0c894533f906b13b79813f02b2982ac675074

- Doc: add example of type resolution in nested UNIONs. Section 10.5 didn't say
explicitly that multiple UNIONs are resolved pairwise. Since the resolution
algorithm is described as taking any number of inputs, readers might well
think that a query like "select x union select y union select z" would be
resolved by considering x, y, and z in one resolution step. But that's not
what happens (and I think that behavior is per SQL spec). Add an example
clarifying this point. Per bug #15129 from Philippe Beaudoin. Discussion:
https://postgr.es/m/152196085023.32649.9916472370480121694@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/c515ff8d0a979fb553136a71388017c97785acda

- Allow memory contexts to have both fixed and variable ident strings.
Originally, we treated memory context names as potentially variable in all
cases, and therefore always copied them into the context header. Commit
9fa6f00b1 rethought this a little bit and invented a distinction between fixed
and variable names, skipping the copy step for the former. But we can make
things both simpler and more useful by instead allowing there to be two parts
to a context's identification, a fixed "name" and an optional, variable
"ident". The name supplied in the context create call is now required to be a
compile-time-constant string in all cases, as it is never copied but just
pointed to. The "ident" string, if wanted, is supplied later. This is needed
because typically we want the ident to be stored inside the context so that
it's cleaned up automatically on context deletion; that means it has to be
copied into the context before we can set the pointer. The cost of this
approach is basically just an additional pointer field in struct
MemoryContextData, which isn't much overhead, and is bought back entirely in
the AllocSet case by not needing a headerSize field anymore, since we no
longer have to cope with variable header length. In addition, we can simplify
the internal interfaces for memory context creation still further, saving a
few cycles there. And it's no longer true that a custom identifier
disqualifies a context from participating in aset.c's freelist scheme, so
possibly there's some win on that end. All the places that were using
non-compile-time-constant context names are adjusted to put the variable info
into the "ident" instead. This allows more effective identification of those
contexts in many cases; for example, subsidary contexts of relcache entries
are now identified by both type (e.g. "index info") and relname, where before
you got only one or the other. Contexts associated with PL function cache
entries are now identified more fully and uniformly, too. I also arranged for
plancache contexts to use the query source string as their identifier. This
is basically free for CachedPlanSources, as they contained a copy of that
string already. We pay an extra pstrdup to do it for CachedPlans. That could
perhaps be avoided, but it would make things more fragile (since the
CachedPlanSource is sometimes destroyed first). I suspect future improvements
in error reporting will require CachedPlans to have a copy of that string
anyway, so it's not clear that it's worth moving mountains to avoid it now.
This also changes the APIs for context statistics routines so that the
context-specific routines no longer assume that output goes straight to
stderr, nor do they know all details of the output format. This is useful
immediately to reduce code duplication, and it also allows for external code
to do something with stats output that's different from printing to stderr.
The reason for pushing this now rather than waiting for v12 is that it
rethinks some of the API changes made by commit 9fa6f00b1. Seems better for
extension authors to endure just one round of API changes not two.
Discussion:
https://postgr.es/m/CAB=Je-FdtmFZ9y9REHD7VsSrnCkiBhsA4mdsLKSPauwXtQBeNA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/442accc3fe0cd556de40d9d6c776449e82254763

- Update pgindent's typedefs blacklist, and make it easier to adjust. It seems
that all buildfarm members are now using the <stdbool.h> code path, so that
none of them report "bool" as a typedef. We still need it to be treated that
way, so adjust pgindent to force that whether or not it's in the given list.
Also, the recent introduction of LLVM infrastructure has caused the appearance
of some typedef names that we definitely *don't* want treated as typedefs,
such as "string" and "abs". Extend the existing blacklist to include these.
(Additions based on comparing v10's typedefs list to what the buildfarm is
currently emitting.) Rearrange the code so that the lists of
whitelisted/blacklisted names are a bit easier to find and modify. Andrew
Dunstan and Tom Lane Discussion:
https://postgr.es/m/28690.1521912334@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/ef1978d6ed1e4defe18d250226460409e6cd5447

- Fix actual and potential double-frees around tuplesort usage.
tuplesort_gettupleslot() passed back tuples allocated in the tuplesort's own
memory context, even when the caller was responsible to free them. This
created a double-free hazard, because some callers might destroy the tuplesort
object (via tuplesort_end) before trying to clean up the last returned tuple.
To avoid this, change the API to specify that the tuple is allocated in the
caller's memory context. v10 and HEAD already did things that way, but in 9.5
and 9.6 this is a live bug that can demonstrably cause crashes with some
grouping-set usages. In 9.5 and 9.6, this requires doing an extra tuple copy
in some cases, which is unfortunate. But the amount of refactoring needed to
avoid it seems excessive for a back-patched change, especially since the cases
where an extra copy happens are less performance-critical. Likewise change
tuplesort_getdatum() to return pass-by-reference Datums in the caller's
context not the tuplesort's context. There seem to be no live bugs among its
callers, but clearly the same sort of situation could happen in future. For
other tuplesort fetch routines, continue to allocate the memory in the
tuplesort's context. This is a little inconsistent with what we now do for
tuplesort_gettupleslot() and tuplesort_getdatum(), but that's preferable to
adding new copy overhead in the back branches where it's clearly unnecessary.
These other fetch routines provide the weakest possible guarantees about tuple
memory lifespan from v10 on, anyway, so this actually seems more consistent
overall. Adjust relevant comments to reflect these API redefinitions.
Arguably, we should change the pre-9.5 branches as well, but since there are
no known failure cases there, it seems not worth the risk. Peter Geoghegan,
per report from Bernd Helmle. Reviewed by Kyotaro Horiguchi; thanks also to
Andreas Seltenreich for extracting a self-contained test case. Discussion:
https://postgr.es/m/1512661638.9720.34.camel@oopsware.de
https://git.postgresql.org/pg/commitdiff/c2d4eb1b1fa252fd8c407e1519308017a18afed1

- While vacuuming a large table, update upper-level FSM data every so often.
VACUUM updates leaf-level FSM entries immediately after cleaning the
corresponding heap blocks. fsmpage.c updates the intra-page search trees on
the leaf-level FSM pages when this happens, but it does not touch the
upper-level FSM pages, so that the released space might not actually be
findable by searchers. Previously, updating the upper-level pages happened
only at the conclusion of the VACUUM run, in a single FreeSpaceMapVacuum()
call. This is bad because the VACUUM might get canceled before ever reaching
that point, so that from the point of view of searchers no space has been
freed at all, leading to table bloat. We can improve matters by updating the
upper pages immediately after each cycle of index-cleaning and heap-cleaning,
processing just the FSM pages corresponding to the range of heap blocks we
have now fully cleaned. This adds a small amount of extra work, since the FSM
pages leading down to each range boundary will be touched twice, but it's
pretty negligible compared to everything else going on in a large VACUUM. If
there are no indexes, VACUUM doesn't work in cycles but just cleans each heap
page on first visit. In that case we just arbitrarily update upper FSM pages
after each 8GB of heap. That maintains the goal of not letting all this work
slide until the very end, and it doesn't seem worth expending extra complexity
on a case that so seldom occurs in practice. In either case, the FSM is fully
up to date before any attempt is made to truncate the relation, so that the
most likely scenario for VACUUM cancellation no longer results in out-of-date
upper FSM pages. When we do successfully truncate, adjusting the FSM to
reflect that is now fully handled within FreeSpaceMapTruncateRel.

- Remove UpdateFreeSpaceMap(), use FreeSpaceMapVacuumRange() instead.
FreeSpaceMapVacuumRange has the same effect, is more efficient if many pages
are involved, and makes fewer assumptions about how it's used. Notably,
Claudio Freire pointed out that UpdateFreeSpaceMap could fail if the specified
freespace value isn't the maximum possible. This isn't a problem for the
single existing user, but the function represents an attractive nuisance IMO,
because it's named as though it were a general-purpose update function and its
limitations are undocumented. In any case we don't need multiple ways to get
the same result. In passing, do some code review and cleanup in
RelationAddExtraBlocks. In particular, I see no excuse for it to omit the
PageIsNew safety check that's done in the mainline extension path in
RelationGetBufferForTuple. Discussion:
https://postgr.es/m/CAGTBQpYR0uJCNTt3M5GOzBRHo+-GccNO1nCaQ8yEJmZKSW5q1A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/a063baaced273e955e088ba5979dcc6ec5cd92e6

- Remove unnecessary BufferGetPage() calls in fsm_vacuum_page(). Just noticed
that these were quite redundant, since we're holding the page address in a
local variable anyway, and we have pin on the buffer throughout. Also improve
a comment.
https://git.postgresql.org/pg/commitdiff/2b1759e2675fc01d6945c9a5fa65c7d7121212f7

- Do index FSM vacuuming sooner. In btree and SP-GiST indexes, move the
responsibility for calling IndexFreeSpaceMapVacuum from the vacuumcleanup
phase to the bulkdelete phase, and do it if and only if we found some pages
that could be put into FSM. As in commit 851a26e26, the idea is to make free
pages visible to FSM searchers sooner when vacuuming very large tables (large
enough to need multiple bulkdelete scans). This adds more redundant work than
that commit did, since we have to scan the entire index FSM each time rather
than being able to localize what needs to be updated; but it still seems
worthwhile. However, we can buy something back by not touching the FSM at all
when there are no pages that can be put in it. That will result in slower
recovery from corrupt upper FSM pages in such a scenario, but it doesn't seem
like that's a case we need to optimize for. Hash indexes don't use FSM at
all. GIN, GiST, and bloom indexes update FSM during the vacuumcleanup phase
not bulkdelete, so that doing something comparable to this would be a much
more invasive change, and it's not clear it's worth it. BRIN indexes do
things sufficiently differently that this change doesn't apply to them,
either. Claudio Freire, reviewed by Masahiko Sawada and Jing Wang, some
additional tweaks by me Discussion:
https://postgr.es/m/CAGTBQpYR0uJCNTt3M5GOzBRHo+-GccNO1nCaQ8yEJmZKSW5q1A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/c79f6df75dd381dbc387326f8155402992524124

- Improve out-of-memory error reports by including memory context name. Add the
target context's name to the errdetail field of "out of memory" errors in
mcxt.c. Per discussion, this seems likely to be useful to help narrow down
the cause of a reported failure, and it costs little. Also, now that context
names are required to be compile-time constants in all cases, there's little
reason to be concerned about security issues from exposing these names to
users. (Because of such concerns, we are *not* including the context "ident"
field.) In passing, add unlikely() markers to the allocation-failed tests,
just to be sure the compiler is on the right page about that. Also, in palloc
and friends, copy CurrentMemoryContext into a local variable, as that's almost
surely cheaper to reference than a global. Discussion:
https://postgr.es/m/1099.1522285628@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/1bb9e731e17b79f5b1c6713159812adfcf4f6495

- Remove obsolete SLRU wrapping and warnings from predicate.c. When SSI was
developed, slru.c was limited to segment files with names in the range
0000-FFFF. This didn't allow enough space for predicate.c to store every
possible XID when spilling old transactions to disk, so it would wrap around
sooner and print warnings. Since commits 638cf09e and 73c986ad increased the
number of segment files slru.c could manage, that behavior is unnecessary.
Therefore remove that code. Also remove the macro OldSerXidSegment, which has
been unused since 4cd3fb6e. Thomas Munro, reviewed by Anastasia Lubennikova
Discussion:
https://postgr.es/m/CAEepm=3XfsTSxgEbEOmxu0QDiXy0o18NUg2nC89JZcCGE+XFPA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/e5eb4fa87331821423b362be5ea4b18e873d5b27

- Ensure that WAL pages skipped by a forced WAL switch are zero-filled. In the
previous coding, skipped pages were mostly zeroes, but they still had valid
WAL page headers. That makes them very much less compressible than an
unbroken string of zeroes would be --- about 10X worse for bzip2 compression,
for instance. We don't need those headers, so tweak the logic so that we zero
them out. Chapman Flack, reviewed by Daniel Gustafsson Discussion:
https://postgr.es/m/579297F8.7020107@anastigmatix.net
https://git.postgresql.org/pg/commitdiff/4a33bb59dfc33566f04e18ab5e1f90b8e7461052

- Fix bogus provolatile/proparallel markings on a few built-in functions.
Richard Yen reported that pg_upgrade failed if the target cluster had
force_parallel_mode = on, because binary_upgrade_create_empty_extension() is
marked parallel restricted, allowing it to be executed in parallel mode, which
complains because it tries to acquire an XID. In general, no function that
might try to modify database data should be considered parallel safe or
restricted, since execution of it might force XID acquisition. We found
several other examples of this mistake. Furthermore, functions that execute
user-supplied SQL queries or query fragments, or pull data from user-supplied
cursors, had better be marked both volatile and parallel unsafe, because we
don't know what the supplied query or cursor might try to do. There were
several tsquery and XML functions that had the wrong proparallel marking for
this, and some of them were even mislabeled as to volatility. All these bugs
are old, dating back to 9.6 for the proparallel mistakes and much further for
the provolatile mistakes. We can't force a catversion bump in the back
branches, but we can at least ensure that installations initdb'd in future
have the right values. Thomas Munro and Tom Lane Discussion:
https://postgr.es/m/CAEepm=2sNDScSLTfyMYu32Q=ob98ZGW-vM_2oLxinzSABGQ6VA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/11002f8afa551f4673aa6a7b62c1872c233e6052

- Fix portability and translatability issues in commit 64f85894a. Compilation
failed for lack of an #ifdef on builds without pg_strong_random(). Also fix
relevant error messages to meet project style guidelines. Fabien Coelho,
further adjusted by me Discussion:
https://postgr.es/m/32390.1522464534@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/5635c7aa678804292fe47bceab5b3f4dae5f39f9

- Fix assorted issues in parallel vacuumdb. Avoid storing the result of
PQsocket() in a pgsocket variable; it's declared as int, and the no-socket
test is properly written as "x < 0" not "x == PGINVALID_SOCKET". This
accidentally had no bad effect because we never got to init_slot() with a bad
connection, but it's still wrong. Actually, it seems like we should avoid
storing the result for a long period at all. The function's not so expensive
that it's worth avoiding, and the existing coding technique here would fail if
anyone tried to PQreset the connection during the life of the program. Hence,
just re-call PQsocket every time we construct a select(2) mask. Speaking of
select(), GetIdleSlot imagined that it could compute the select mask once and
continue to use it over multiple calls to select_loop(), which is pretty bogus
since that would stomp on the mask on return. This could only matter if the
function's outer loop iterated more than once, which is unlikely (it'd take
some connection receiving data, but not enough to complete its command). But
if it did happen, we'd acquire "tunnel vision" and stop watching the other
connections for query termination, with the effect of losing parallelism.
Another way in which GetIdleSlot could lose parallelism is that once PQisBusy
returns false, it would lock in on that connection and do PQgetResult until
that returns NULL; in some cases that could result in blocking. (Perhaps this
can never happen in vacuumdb due to the limited set of commands that it can
issue, but I'm not quite sure of that, and even if true today it's not a
future-proof assumption.) Refactor the code to do that properly, so that it
risks blocking in PQgetResult only in cases where we need to wait anyway.
Another loss-of-parallelism problem, which *is* easily demonstrable, is that
any setup queries issued during prepare_vacuum_command() were always issued on
the last-to-be-created connection, whether or not that was idle. Long-running
operations on that connection thus prevented issuance of additional operations
on the other ones, except in the limited cases where no preparatory query was
needed. Instead, wait till we've identified a free connection and use that
one. Also, avoid core dump due to undersized malloc request in the case that
no tables are identified to be vacuumed. The bogus no-socket test was noted
by CharSyam, the other problems identified in my own code review. Back-patch
to 9.5 where parallel vacuumdb was introduced. Discussion:
https://postgr.es/m/CAMrLSE6etb33-192DTEUGkV-TsvEcxtBDxGWG1tgNOMnQHwgDA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/94173d3eeb445ae49020c5e2d9e166856d923eaf

- Portability fix for commit 9a895462d. So far as I can find, NI_MAXHOST isn't
actually required anywhere by POSIX. Nonetheless, commit 9a895462d supposed
that it could rely on having that symbol without any ceremony at all. We do
have a hack for providing it if the platform doesn't, in getaddrinfo.h, so fix
the problem by #including that file. Per buildfarm.
https://git.postgresql.org/pg/commitdiff/b0c90c85fc93f37107365dd8c7f47ba5e00544de

- Fix a boatload of typos in C comments. Justin Pryzby Discussion:
https://postgr.es/m/20180331105640.GK28454@telsasoft.com
https://git.postgresql.org/pg/commitdiff/0b11a674fb11cc1571326c861ecdd7773d9e587f

Andrew Dunstan pushed:

- Remove two tests inadvertently added in 2b27273435.
https://git.postgresql.org/pg/commitdiff/1d494b622fa9dfdbd7213f357cd38def0125a322

- Make fast_default regression tests locale independent.
https://git.postgresql.org/pg/commitdiff/a437551a228a5099c305d1376188d6926c043724

- Fast ALTER TABLE ADD COLUMN with a non-NULL default. Currently adding a
column to a table with a non-NULL default results in a rewrite of the table.
For large tables this can be both expensive and disruptive. This patch removes
the need for the rewrite as long as the default value is not volatile. The
default expression is evaluated at the time of the ALTER TABLE and the result
stored in a new column (attmissingval) in pg_attribute, and a new column
(atthasmissing) is set to true. Any existing row when fetched will be supplied
with the attmissingval. New rows will have the supplied value or the default
and so will never need the attmissingval. Any time the table is rewritten all
the atthasmissing and attmissingval settings for the attributes are cleared,
as they are no longer needed. The most visible code change from this is in
heap_attisnull, which acquires a third TupleDesc argument, allowing it to
detect a missing value if there is one. In many cases where it is known that
there will not be any (e.g. catalog relations) NULL can be passed for this
argument. Andrew Dunstan, heavily modified from an original patch from Serge
Rielau. Reviewed by Tom Lane, Andres Freund, Tomas Vondra and David Rowley.
Discussion:
https://postgr.es/m/31e2e921-7002-4c27-59f5-51f08404c858@2ndQuadrant.com
https://git.postgresql.org/pg/commitdiff/16828d5c0273b4fe5f10f42588005f16b415b2d8

- Optimize btree insertions for common case of increasing values. Remember the
last page of an index insert if it's the rightmost leaf page. If the next
entry belongs on and can fit in the remembered page, insert the new entry
there as long as we can get a lock on the page. Otherwise, fall back on the
more expensive method of searching for the right place to insert the entry.
This provides a performance improvement for the common case where an index
entry is for monotonically increasing or nearly monotonically increasing value
such as an identity field or a current timestamp. Pavan Deolasee Reviewed by
Claudio Freire, Simon Riggs and Peter Geoghegan Discussion:
https://postgr.es/m/CABOikdM9DrupjyKZZFM5k8-0RCDs1wk6JzEkg7UgSW6QzOwMZw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2b27273435392d1606f0ffc95d73a439a457f08e

- Small cleanups in fast default code. Problems identified by Andres Freund and
Haribabu Kommi
https://git.postgresql.org/pg/commitdiff/ed69864350a59c51c8570900601ebd335956b638

Álvaro Herrera pushed:

- Fix typo.
https://git.postgresql.org/pg/commitdiff/1b89c2188bd38eac68251f16051859996128f2d0

- Handle INSERT .. ON CONFLICT with partitioned tables. Commit eb7ed3f30634
enabled unique constraints on partitioned tables, but one thing that was not
working properly is INSERT/ON CONFLICT. This commit introduces a new node
keeps state related to the ON CONFLICT clause per partition, and fills it when
that partition is about to be used for tuple routing. Author: Amit Langote,
Álvaro Herrera Reviewed-by: Etsuro Fujita, Pavan Deolasee Discussion:
https://postgr.es/m/20180228004602.cwdyralmg5ejdqkq@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/555ee77a9668e3f1b03307055b5027e13bf1a715

- Fix test impredictability. Test 'triggers' fails when another one creates
triggers concurrently at some precise time, because of a missing WHERE clause.
Per buildfarm members snapper, desmoxytes.
https://git.postgresql.org/pg/commitdiff/186b6df2e62251e5e1f3cae8a3257c7226f4188c

- Fix thinko in comment. The listed numbers disagreed with the ones being used
in the symbols; but instead of just fixing the numbers in the comment, use the
symbolic name instead, which seems clearer. This has been wrong all along, so
apply back to 9.5 where BRIN was introduced. Reported-by: Tomas Vondra
Discussion:
https://postgr.es/m/5ff514f2-8b1e-6366-b11c-8e2ed442562d@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/530bcf7581574d5b0f26c2eaeef1c32bbcd37907

Andres Freund pushed:

- Correct some typos in the new JIT code. Author: Thomas Munro
https://git.postgresql.org/pg/commitdiff/96b5eac9186e033c67944124803ef5aa8f246afc

- JIT tuple deforming in LLVM JIT provider. Performing JIT compilation for
deforming gains performance benefits over unJITed deforming from compile-time
knowledge of the tuple descriptor. Fixed column widths, NOT NULLness, etc can
be taken advantage of. Right now the JITed deforming is only used when
deforming tuples as part of expression evaluation (and obviously only if the
descriptor is known). It's likely to be beneficial in other cases, too. By
default tuple deforming is JITed whenever an expression is JIT compiled.
There's a separate boolean GUC controlling it, but that's expected to be
primarily useful for development and benchmarking. Docs will follow in a
later commit containing docs for the whole JIT feature. Author: Andres Freund
Discussion:
https://postgr.es/m/20170901064131.tazjxwus3k2w3ybh@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/32af96b2b118cd204ca809d7c48c7f8ea7f879cf

- LLVMJIT: Free created module in LLVM < 5. Due to the differing APIs between
versions, I forgot to deallocate the generated module in older LLVM versions,
leading to a memory leak. Author: Andres Freund
https://git.postgresql.org/pg/commitdiff/071371bc43c89d6db923a7f858933f655b150655

- Make new regression indpendent of max_parallel_workers_per_gather. The tests
in e2f1eb0ee30d1 ("Implement partition-wise grouping/aggregation.") weren't
independent of the server's max_parallel_workers_per_gather setting. I
(Andres) find it useful to locally run with that disabled, and the
aforementioned patch broke this. Author: Jeevan Chalke Discussion:
https://postgr.es/m/20180322210703.qmga3vsxqmiiypci@alap3.anarazel.de
https://postgr.es/m/CAM2+6=UNWGKTgh9aOn4=SQ72HfFzbVFseh9=5N54bD6KB+D9OQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/0976c4ddd4c98d64b2f3140ee96b92367b763a44

- Adapt to LLVM 7+ Orc API changes. This is mostly done to be able to validate
features and fixes submitted to LLVM. Given the size of these changes that
seems acceptable. Author: Andres Freund
https://git.postgresql.org/pg/commitdiff/4b9094eb6e14dfdbed61278ea8e51cc846e43579

- Add catversion bump missed in 16828d5c0. Given that pg_attribute changed its
layout...
https://git.postgresql.org/pg/commitdiff/b4013b8e4a271816ba87aa56f46dbc04a083d962

- Quick adaption of JIT tuple deforming to the fast default patch. Instead
using memset to set tts_isnull, call the new slot_getmissingattrs(). Also fix
a bug (= instead of >=) in the code generation. Normally = is correct, but
when repeatedly deforming fields not in a tuple (e.g. deform up to natts + 1
and then natts + 2) >= is needed. Discussion:
https://postgr.es/m/20180328010053.i2qvsuuusst4lgmc@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/f4f5845b3182ab930e525b1419bca47ac611604e

- Use isinf builtin for clang, for performance. When compiling with clang
glibc's definition of isinf() ends up leading to and external libc function
call. That's because there was a bug in the builtin in an old gcc version, and
clang claims compatibility with an older version. That causes clang to be
measurably slower for floating point heavy workloads than gcc. To fix simply
redirect isinf when using clang and clang confirms it has __builtin_isinf().
https://git.postgresql.org/pg/commitdiff/8a934d6778331f2ac04a40f4f22178a56a232315

- Add inlining support to LLVM JIT provider. This provides infrastructure to
allow JITed code to inline code implemented in C. This e.g. can be postgres
internal functions or extension code. This already speeds up long running
queries, by allowing the LLVM optimizer to optimize across function
boundaries. The optimization potential currently doesn't reach its full
potential because LLVM cannot optimize the FunctionCallInfoData argument fully
away, because it's allocated on the heap rather than the stack. Fixing that is
beyond what's realistic for v11. To be able to do that, use CLANG to convert
C code to LLVM bitcode, and have LLVM build a summary for it. That bitcode can
then be used to to inline functions at runtime. For that the bitcode needs to
be installed. Postgres bitcode goes into $pkglibdir/bitcode/postgres,
extensions go into equivalent directories. PGXS has been modified so that
happens automatically if postgres has been compiled with LLVM support.
Currently this isn't the fastest inline implementation, modules are reloaded
from disk during inlining. That's to work around an apparent LLVM bug,
triggering an apparently spurious error in LLVM assertion enabled builds.
Once that is resolved we can remove the superfluous read from disk. Docs will
follow in a later commit containing docs for the whole JIT feature. Author:
Andres Freund Discussion:
https://postgr.es/m/20170901064131.tazjxwus3k2w3ybh@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/9370462e9a79755aea367c62eb0fef96f0c42258

- Add EXPLAIN support for JIT. This just shows a few details about JITing, e.g.
how many functions have been JITed, and how long that took. To avoid noise in
regression tests with functions sometimes being JITed in --with-llvm builds,
disable display when COSTS OFF is specified. Author: Andres Freund
Discussion:
https://postgr.es/m/20170901064131.tazjxwus3k2w3ybh@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/1f0c6a9e7dca70ba7d2c949e42298d764ca457c0

- Fix mistakes in the just added JIT docs. Reported-By: Lukas Fittl Author:
Andres Freund
https://git.postgresql.org/pg/commitdiff/a0a08c1d85dae3c332e7d72a56df0636be9c5d0a

- Improve JIT docs. Author: John Naylor and Andres Freund Discussion:
https://postgr.es/m/CAJVSVGUs-VcwSY7-Kx-GQe__8hvWuA4Uhyf3gxoMXeiZqebE9g@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/fb604780114cea6a83f3f6a60e7f51a7185c932b

- Combine options for RangeVarGetRelidExtended() into a flags argument. A
followup patch will add a SKIP_LOCKED option. To avoid introducing evermore
arguments, breaking existing callers each time, introduce a flags argument.
This'll no doubt break a few external users... Also change the MISSING_OK
behaviour so a DEBUG1 debug message is emitted when a relation is not found.
Author: Nathan Bossart Reviewed-By: Michael Paquier and Andres Freund
Discussion:
https://postgr.es/m/20180306005349.b65whmvj7z6hbe2y@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/d87510a524f36a630cfb34cc392e95e959a1b0dc

- Add SKIP_LOCKED option to RangeVarGetRelidExtended(). This will be used for
VACUUM (SKIP LOCKED). Author: Nathan Bossart Reviewed-By: Michael Paquier and
Andres Freund Discussion:
https://postgr.es/m/20180306005349.b65whmvj7z6hbe2y@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/3e256e550672657375fc3058b2b8ff6568d65cef

- Remove PARTIAL_LINKING build mode. In
9956ddc19164b02dc1925fb389a1af77472eba5e, ten years ago, the current
objfile.txt based linking model was introduced. It's time to retire the old
SUBSYS.o based model. This primarily is pertinent because the bitcode files
for LLVM based inlining are not produced when using PARTIAL_LINKING. It does
not seem worth to fix PARTIAL_LINKING to support that. Author: Andres Freund
Discussion:
https://postgr.es/m/20180121204356.d5oeu34jetqhmdv2@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/a4ebbd27527087fcf3ade36f4e0072033e5b2f78

- Add documentation for the JIT feature. As promised in earlier commits, this
adds documentation about the new build options, the new GUCs, about the
planner logic when JIT is used, and the benefits of JIT in general. Also adds
a more implementation oriented README. I'm sure we're going to want to expand
this further, but I think this is a reasonable start. Author: Andres Freund,
with contributions by Thomas Munro Reviewed-By: Thomas Munro Discussion:
https://postgr.es/m/20170901064131.tazjxwus3k2w3ybh@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/e6c039d13e16a3a2dec5ba479d9d1fb3229c03a3

- Add Bloom filter implementation. A Bloom filter is a space-efficient,
probabilistic data structure that can be used to test set membership. Callers
will sometimes incur false positives, but never false negatives. The rate of
false positives is a function of the total number of elements and the amount
of memory available for the Bloom filter. Two classic applications of Bloom
filters are cache filtering, and data synchronization testing. Any user of
Bloom filters must accept the possibility of false positives as a cost worth
paying for the benefit in space efficiency. This commit adds a test harness
extension module, test_bloomfilter. It can be used to get a sense of how the
Bloom filter implementation performs under varying conditions. This is
infrastructure for the upcoming "heapallindexed" amcheck patch, which verifies
the consistency of a heap relation against one of its indexes. Author: Peter
Geoghegan Reviewed-By: Andrey Borodin, Michael Paquier, Thomas Munro, Andres
Freund Discussion:
https://postgr.es/m/CAH2-Wzm5VmG7cu1N-H=nnS57wZThoSDQU+F5dewx3o84M+jY=g@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/51bc271790eb234a1ba4d14d3e6530f70de92ab5

- Add amcheck verification of heap relations belonging to btree indexes. Add a
new, optional, capability to bt_index_check() and bt_index_parent_check():
check that each heap tuple that should have an index entry does in fact have
one. The extra checking is performed at the end of the existing nbtree
checks. This is implemented by using a Bloom filter data structure. The
implementation performs set membership tests within a callback (the same type
of callback that each index AM registers for CREATE INDEX). The Bloom filter
is populated during the initial index verification scan. Reusing the CREATE
INDEX infrastructure allows the new verification option to automatically
benefit from the heap consistency checks that CREATE INDEX already performs.
CREATE INDEX does thorough sanity checking of HOT chains, so the new check
actually manages to detect problems in heap-only tuples. Author: Peter
Geoghegan Reviewed-By: Pavan Deolasee, Andres Freund Discussion:
https://postgr.es/m/CAH2-Wzm5VmG7cu1N-H=nnS57wZThoSDQU+F5dewx3o84M+jY=g@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7f563c09f8901f6acd72cb8fba7b1bd3cf3aca8e

- Fix non-portable use of round(). round() is from C99. Use rint() instead.
There are behavioral differences between round() and rint(), but they should
not matter to the Bloom filter optimal_k() function. We already assume POSIX
behavior for rint(), so there is no question of rint() not using "rounds
towards nearest" as its rounding mode. Cleanup from commit
51bc271790eb234a1ba4d14d3e6530f70de92ab5. Per buildfarm member thrips.
Author: Peter Geoghegan Discussion:
https://postgr.es/m/CAH2-Wzn76eCGUonARy-wrVtMHsf+4cvbK_oJAWTLfORTU5ki0w@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/686d399f2be6eea4b74d59cdadd427d09cb0e246

Teodor Sigaev pushed:

- Add predicate locking for GiST. Add page-level predicate locking, due to
gist's code organization, patch seems close to trivial: add check before page
changing, add predicate lock before page scanning. Although choosing right
place to check is not simple: it should not be called during index build, it
should support insertion of new downlink and so on. Author: Shubham Barai
with editorization by me and Alexander Korotkov Reviewed by: Alexander
Korotkov, Andrey Borodin, me Discussion:
https://www.postgresql.org/message-id/flat/CALxAEPtdcANpw5ePU3LvnTP8HCENFw6wygupQAyNBgD-sG3h0g(at)mail(dot)gmail(dot)com
https://git.postgresql.org/pg/commitdiff/3ad55863e9392bff73377911ebbf9760027ed405

- Fix count of skipped test of basebackup on Windows. Commit
920a5e500a119b03356fb1fb64a677eb1aa5fc6f add tests which should be skipped on
Windows boxes, but patch doesn't contain right count of them. David Steel
https://git.postgresql.org/pg/commitdiff/44bd95846a526dd1f69bdc78b3832f2d2de77dd4

- Skip temp tables from basebackup. Do not store temp tables in basebackup,
they will not be visible anyway, so, there are not reasons to store them.
Author: David Steel Reviewed by: me Discussion:
https://www.postgresql.org/message-id/flat/5ea4d26a-a453-c1b7-eff9-5a3ef8f8aceb(at)pgmasters(dot)net
https://git.postgresql.org/pg/commitdiff/920a5e500a119b03356fb1fb64a677eb1aa5fc6f

- Add casts from jsonb. Add explicit cast from scalar jsonb to all numeric and
bool types. It would be better to have cast from scalar jsonb to text too but
there is already a cast from jsonb to text as just text representation of
json. There is no way to have two different casts for the same type's pair.
Bump catalog version Author: Anastasia Lubennikova with editorization by
Nikita Glukhov and me Review by: Aleksander Alekseev, Nikita Glukhov, Darafei
Praliaskouski Discussion:
https://www.postgresql.org/message-id/flat/0154d35a-24ae-f063-5273-9ffcdf1c7f2e(at)postgrespro(dot)ru
https://git.postgresql.org/pg/commitdiff/c0cbe00fee6d0a5e0ec72c6d68a035e674edc4cc

- Predicate locking in GIN index. Predicate locks are used on per page basis
only if fastupdate = off, in opposite case predicate lock on pending list will
effectively lock whole index, to reduce locking overhead, just lock a
relation. Entry and posting trees are essentially B-tree, so locks are
acquired on leaf pages only. Author: Shubham Barai with some editorization by
me and Dmitry Ivanov Review by: Alexander Korotkov, Dmitry Ivanov, Fedor
Sigaev Discussion:
https://www.postgresql.org/message-id/flat/CALxAEPt5sWW+EwTaKUGFL5_XFcZ0MuGBcyJ70oqbWqr42YKR8Q(at)mail(dot)gmail(dot)com
https://git.postgresql.org/pg/commitdiff/43d1ed60fdd96027f044e152176c0d45cd6bf443

- Set random seed for pgbench. Setting random could increase reproducibility of
test in some cases. Patch suggests three providers for seed: time (default),
strong random generator (if available) and unsigned constant. Seed could be
set from command line or enviroment variable. Author: Fabien Coelho Reviewed
by: Chapman Flack Discussion:
https://www.postgresql.org/message-id/flat/20160407082711(dot)q7iq3ykffqxcszkv(at)alap3(dot)anarazel(dot)de
https://git.postgresql.org/pg/commitdiff/64f85894ad2730fb1449a8e81dd8026604e9a546

Simon Riggs pushed:

- Use pg_stat_get_xact* functions within xacts. Resolve build farm failures
from c203d6cf81b4d7e43, diagnosed by Tom Lane. The output of
pg_stat_get_xact_tuples_hot_updated() and friends is not guaranteed to show
anything after the transaction completes. Data is flushed slowly to stats
collector, so using them can give timing issues.
https://git.postgresql.org/pg/commitdiff/5b0d7f6996abfc1e3e51bac62af6076903635dc8

- Allow HOT updates for some expression indexes. If the value of an index
expression is unchanged after UPDATE, allow HOT updates where previously we
disallowed them, giving a significant performance boost in those cases.
Particularly useful for indexes such as JSON->>field where the JSON value
changes but the indexed value does not. Submitted as "surjective indexes"
patch, now enabled by use of new "recheck_on_update" parameter. Author:
Konstantin Knizhnik Reviewer: Simon Riggs, with much wordsmithing and some
cleanup
https://git.postgresql.org/pg/commitdiff/c203d6cf81b4d7e43edb2b75ec1b741ba48e04e0

- Store 2PC GID in commit/abort WAL recs for logical decoding. Store GID of 2PC
in commit/abort WAL records when wal_level = logical. This allows logical
decoding to send the SAME gid to subscribers across restarts of logical
replication. Track replica origin replay progress for 2PC. (Edited from patch
0003 in the logical decoding 2PC series.) Authors: Nikhil Sontakke, Stas
Kelvich Reviewed-by: Simon Riggs, Andres Freund
https://git.postgresql.org/pg/commitdiff/1eb6d6527aae264b3e0b9c95aa70bb7a594ad1cf

Fujii Masao pushed:

- Fix handling of files that source server removes during pg_rewind is running.
After processing the filemap to build the list of chunks that will be fetched
from the source to rewing the target server, it is possible that a file which
was previously processed is removed from the source. A simple example of such
an occurence is a WAL segment which gets recycled on the target in-between.
When the filemap is processed, files not categorized as relation files are
first truncated to prepare for its full copy of which is going to be taken
from the source, divided into a set of junks. However, for a recycled WAL
segment, this would result in a segment which has a zero-byte size. With such
an empty file, post-rewind recovery thinks that records are saved but they are
actually not because of the truncation which happened when processing the
filemap, resulting in data loss. In order to fix the problem, make sure that
files which are found as removed on the source when receiving chunks of them
are as well deleted on the target server for consistency. Back-patch to 9.5
where pg_rewind was added. Author: Tsunakawa Takayuki Reviewed-by: Michael
Paquier Reported-by: Tsunakawa Takayuki Discussion:
https://postgr.es/m/0A3221C70F24FB45833433255569204D1F8DAAA2%40G01JPEXMBYT05
https://git.postgresql.org/pg/commitdiff/09e96b3f35627a2939e2effd8b98aaa934f59b32

- Make pg_rewind skip files and directories that are removed during server
start. The target cluster that was rewound needs to perform recovery from the
checkpoint created at failover, which leads it to remove or recreate some
files and directories that may have been copied from the source cluster. So
pg_rewind can skip synchronizing such files and directories, and which reduces
the amount of data transferred during a rewind without changing the usefulness
of the operation. Author: Michael Paquier Reviewed-by: Anastasia Lubennikova,
Stephen Frost and me Discussion:
https://postgr.es/m/20180205071022.GA17337@paquier.xyz
https://git.postgresql.org/pg/commitdiff/266b6acb312fc440c1c1a2036aa9da94916beac6

- Enhance pg_stat_wal_receiver view to display host and port of sender server.
Previously there was no way in the standby side to find out the host and port
of the sender server that the walreceiver was currently connected to when
multiple hosts and ports were specified in primary_conninfo. For that purpose,
this patch adds sender_host and sender_port columns into pg_stat_wal_receiver
view. They report the host and port that the active replication connection
currently uses. Bump catalog version. Author: Haribabu Kommi Reviewed-by:
Michael Paquier and me Discussion:
https://postgr.es/m/CAJrrPGcV_aq8=cdqkFhVDJKEnDQ70yRTTdY9RODzMnXNrCz2Ow@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/9a895462d940c0694042059f90e5f63a0a517ded

Bruce Momjian pushed:

- C comment: fix typo, log -> lag. Reported-by: atorikoshi Discussion:
https://postgr.es/m/b61f2ab9-c0e0-d33d-ce3f-42a228025681@lab.ntt.co.jp Author:
atorikoshi
https://git.postgresql.org/pg/commitdiff/a2894cce544d120199a1a90469073796d055bb60

- docs: fix INSTALL.xml build by using "standalone-ignore". Was broken by
"jit" link.
https://git.postgresql.org/pg/commitdiff/7fe04ce9203cb0c5332614ec091aab28cf6aeaa8

- C comment: fix wording about shared memory message queue. Reported-by: Tels
Discussion:
https://postgr.es/m/e66e05bc55f5ce904e361ad17a3395ae.squirrel@sm.webmail.pair.com
https://git.postgresql.org/pg/commitdiff/bc0021ef09ec709fa20309228ea30ccf07f8b4e6

- README change: update for hash access method. Reported-by: Thomas Munro,
Justin Pryzby Discussion:
https://postgr.es/m/CAEepm=1_682z-09DNHj4GkCJAqWK-D6h9Oq5ea84T1oqq1-Utg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3282c4c136e4e5ad22d57dbe7a98fbac2962500a

- C comments: "a" <--> "an" corrections. Reported-by: Michael Paquier, Abhijit
Menon-Sen Discussion: https://postgr.es/m/20180305045854.GB2266@paquier.xyz
Author: Michael Paquier, Abhijit Menon-Sen, me
https://git.postgresql.org/pg/commitdiff/20b4323bd107920a3c3e60452442e8e2cee302d2

- docs: fix spacing around "if not exists" brackets. Reported-by: Fabrízio de
Royes Mello Discussion:
https://postgr.es/m/CAFcNs+qDD+QKcF8YCPQnjAxoWN61qY_YdFLB3iQqbWCLSCyY0g@mail.gmail.com
Author: Fabrízio de Royes Mello
https://git.postgresql.org/pg/commitdiff/3da7502cd00ddf8228c9a4a7e4a08725decff99c

- doc: document "IS NOT DOCUMENT". Reported-by: scott(dot)ure(at)caseware(dot)com
Discussion:
https://postgr.es/m/152056505045.4963.16783351661813640274@wrigleys.postgresql.org
Author: Euler Taveira Backpatch-through: 10
https://git.postgresql.org/pg/commitdiff/756dca8e7f65edb479b7e94b024840011f07565c

- docs: add parameter with brackets around varbit(). Reported-by:
scott(dot)ure(at)caseware(dot)com Discussion:
https://postgr.es/m/152074343671.1853.18284519607571497106@wrigleys.postgresql.org
Author: Euler Taveira Backpatch-through: 10
https://git.postgresql.org/pg/commitdiff/681673e0c6c21a876e6ef7680e2116de60cf6286

Magnus Hagander pushed:

- Fix typo in comment. Arthur Zakirov, confirmed by Thomas Munro
https://git.postgresql.org/pg/commitdiff/669820a3d9c359e8d44a26035cb4d675dc542cb4

- Fix typo in comment. Author: Daniel Gustafsson <daniel(at)yesql(dot)se>
https://git.postgresql.org/pg/commitdiff/9778d5c180c06998c315a4ae4753b927e3bc4ea8

- Fix incorrect copy/paste in comment. Author: Alexander Korotkov
<a(dot)korotkov(at)postgrespro(dot)ru>
https://git.postgresql.org/pg/commitdiff/8cdc834647b8b1558c10a7d27a3580a32e04c500

- Fix typo in comment. Author: Michael Paquier <michael(at)paquier(dot)xyz>
https://git.postgresql.org/pg/commitdiff/019fa576ca8298ecb7b8ded6e0c857840b57a4ae

Robert Haas pushed:

- Teach create_projection_plan to omit projection where possible. We sometimes
insert a ProjectionPath into a plan tree when projection is not strictly
required. The existing code already arranges to avoid emitting a Result node
when the ProjectionPath's subpath can perform the projection itself, but
previously it didn't consider the possibility that the parent node might not
actually require the projection to be performed at all. Skipping projection
when it's not required can not only avoid Result nodes that aren't needed, but
also avoid losing the "physical tlist" optimization unneccessarily. Patch by
me, reviewed by Amit Kapila. Discussion:
http://postgr.es/m/CA+TgmoakT5gmahbPWGqrR2nAdFOMAOnOXYoWHRdVfGWs34t6_A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d7c19e62a8e0a634eb6b29f8f1111d944e57081f

- Postpone generate_gather_paths for topmost scan/join rel. Don't call
generate_gather_paths for the topmost scan/join relation when it is initially
populated with paths. Instead, do the work in grouping_planner. By itself,
this gains nothing; in fact it loses slightly because we end up calling
set_cheapest() for the topmost scan/join rel twice rather than once. However,
it paves the way for a future commit which will postpone generate_gather_paths
for the topmost scan/join relation even further, allowing more accurate
costing of parallel paths. Amit Kapila and Robert Haas. Earlier versions of
this patch (which different substantially) were reviewed by Dilip Kumar, Amit
Khandekar, Marina Polyakova, and Ashutosh Bapat.
https://git.postgresql.org/pg/commitdiff/3f90ec8597c3515e0d3190613b31491686027e4b

- Remove 'target' from GroupPathExtraData. It's not needed. Jeevan Chalke
Discussion:
http://postgr.es/m/CAM2+6=XPWujjmj5zUaBTGDoB38CemwcPmjkRy0qOcsQj_V+2sQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/c1de1a3a8b93a61e8264484a10a482156026d12c

- Rewrite the code that applies scan/join targets to paths. If the toplevel
scan/join target list is parallel-safe, postpone generating Gather (or Gather
Merge) paths until after the toplevel has been adjusted to return it. This
(correctly) makes queries with expensive functions in the target list more
likely to choose a parallel plan, since the cost of the plan now reflects the
fact that the evaluation will happen in the workers rather than the leader.
The original complaint about this problem was from Jeff Janes. If the
toplevel scan/join relation is partitioned, recursively apply the changes to
all partitions. This sometimes allows us to get rid of Result nodes, because
Append is not projection-capable but its children may be. It also cleans up
what appears to be incorrect SRF handling from commit
e2f1eb0ee30d144628ab523432320f174a2c8966: the old code had no knowledge of
SRFs for child scan/join rels. Because we now use create_projection_path() in
some cases where we formerly used apply_projection_to_path(), this changes the
ordering of columns in some queries generated by postgres_fdw. Update
regression outputs accordingly. Patch by me, reviewed by Amit Kapila and by
Ashutosh Bapat. Other fixes for this problem (substantially different from
this version) were reviewed by Dilip Kumar, Amit Khandekar, and Marina
Polyakova. Discussion:
http://postgr.es/m/CAMkU=1ycXNipvhWuweUVpKuyu6SpNjF=yHWu4c4US5JgVGxtZQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/11cf92f6e2e13c0a6e3f98be3e629e6bd90b74d5

- Don't call IS_DUMMY_REL() when cheapest_total_path might be junk. Unlike the
previous coding, this might result in a Gather per Append subplan when the
target list is parallel-restricted, but such a plan is probably worth
considering in that case, since a single Gather on top of the entire Append is
impossible. Per Andres Freund and the buildfarm. Discussion:
http://postgr.es/m/20180330050351.bmxx4cdtz67czjda@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/96030f9a481a78483945447e122a387902a4549b

Tatsuo Ishii pushed:

- Allow to lock views. Now all tables used in view definitions can be
recursively locked by a LOCK command. Author: Yugo Nagata Reviewed by Robert
Haas, Thomas Munro and me. Discussion:
https://postgr.es/m/20171011183629.eb2817b3.nagata%40sraoss.co.jp
https://git.postgresql.org/pg/commitdiff/34c20de4d0b0ea8f96d0c518724d876c7b984cf5

- Fix bug with view locking code. LockViewRecurese() obtains view relation
using heap_open() and passes it to get_view_query() to get view info. It
immediately closes the relation then uses the returned view info by calling
LockViewRecurse_walker(). Since get_view_query() returns a pointer within the
relcache, the relcache should be kept until LockViewRecurse_walker() returns.
Otherwise the relation could point to a garbage memory area. Fix is moving
the heap_close() call after LockViewRecurse_walker(). Problem reported by Tom
Lane (buildfarm is unhappy, especially prion since it enables
-DRELCACHE_FORCE_RELEASE cpp flag), fix by me.
https://git.postgresql.org/pg/commitdiff/1b26bd4089a388929c644ffea2832f3841c25969

== Pending Patches ==

Kyotaro HORIGUCHI sent in another revision of a patch to fix a bug in ECPG where
freeing memory for pgtypes crashes on Windows.

Craig Ringer sent in a patch to add pg_xact_commit_timestamp_origin().

Pavan Deolasee sent in a patch to fix and off-by-one error in basebackup.c.

Nikita Glukhov sent in another revision of a patch to add missing type
conversion functions for PL/PythonU.

Amit Langote sent in two more revisions of a patch to implement ON CONFLICT DO
UPDATE for partitioned tables.

Doug Rady sent in another revision of a patch to add an option to pgbench to
build using ppoll() for larger connection counts.

Peter Eisentraut sent in another revision of a patch to pL/pgSQL to allow
committing inside cursor loop.

Pavan Deolasee sent in three more revisions of a patch to implement MERGE.

Amit Langote, David Rowley, and Jesper Pedersen traded patches to speed up
partition pruning.

Ildus Kurbangaliev sent in two more revisions of a patch to implement custom
compression.

David Rowley and Tom Lane traded patches to add parallel aggregates for
string_agg and array_agg.

John Naylor sent in three more revisions of a patch to rationalize the treatment
of bootstrap data.

Robert Haas and Amit Kapila traded patches to enable parallel seq scan for slow
functions.

Pavel Stěhule sent in two more revisions of a patch to add a csv output format
to psql.

Stephen Frost sent in two more revisions of a patch to add a
pg_access_server_files default role.

Thomas Munro, Dmitry Ivanov, and Aleksandr Parfenov traded patches to add a
websearch_to_tsquery() function.

Fabien COELHO sent in another revision of a patch to add a function to pgbench to
test for existence of a variable.

Daniel Gustafsson sent in two revisions of a patch to ensure that maxlen is an
integer value in dict_int configuration.

Markus Winand sent in a patch to accept TEXT and CDATA nodes in XMLTABLE's
column_expression and set the correct context for XPath evaluation.

Nikita Glukhov sent in another revision of a patch to implement JSONPATH.

Etsuro Fujita sent in a patch to fix an issue where COPY was not looking at a
partition's constraints and could result in having added non-compliant data to
it.

Edmund Horner sent in a patch to allocate enough shared string memory for the
stats of auxiliary processes.

Jeevan Chalke sent in two more revisions of a patch to fix partition-wise
aggregation/grouping.

Fabien COELHO sent in a patch to fix some typos in the pgbench docs.

Marina Polyakova sent in two more revisions of a patch to fix pgbench errors and
serialization/deadlock retries.

David Steele sent in two more revisions of a patch to implement a configurable
file mode mask.

Thomas Munro sent in two revisions of a patch to mark
binary_upgrade_create_empty_extension PARALLEL UNSAFE.

Michaël Paquier and Kyotaro HORIGUCHI traded patches to fix an issue with
setting the full page writes using SIGHUP.

Tomáš Vondra sent in three more revisions of a patch to implement multivariate
histograms and MCV lists.

Konstantin Knizhnik sent in another revision of a patch to allow HOT updates for
some expression indexes.

Amit Langote sent in another revision of a patch to reorganize the partitioning
code.

Konstantin Knizhnik sent in three revisions of a patch to implement libpq
compression.

Dmitry Ivanov, Teodor Sigaev, and Alexander Korotkov traded patches to implement
predicate locking for GIN indexes.

Alexander Kuzmenkov and Tomáš Vondra traded patches to implement incremental
sort.

Artur Zakirov sent in two more revisions of a patch to enable shared ISpell
dictionaries.

Anastasia Lubennikova and Alexander Korotkov traded patches to implement
covering + unique indexes, and Andrey Borodin added tests for same.

Ildus Kurbangaliev sent in another revision of a patch to implement a prefix
operator for text and spgist support.

Haribabu Kommi sent in another revision of a patch to implement pluggable
storage.

Daniel Gustafsson sent in another revision of a patch to support an optional
message in backend cancel/terminate.

Peter Eisentraut sent in another revision of a patch to implement SET
TRANSACTION in PL/pgsql.

Jesper Pedersen sent in a patch to clarify the JIT README.

Craig Ringer sent in a patch to fix an issue where PostgreSQL's handling of
fsync() errors is unsafe and risks data loss at least on XFS.

Kyotaro HORIGUCHI sent in another revision of a patch to fix a WAL logging
issue.

Kyotaro HORIGUCHI and Amit Langote traded patches to fix a broken ATTACH
PARTITION behavior.

Etsuro Fujita sent in another revision of a patch to add tuple routing for
foreign partitions.

Fujii Masao sent in a patch to speed up relation deletes during recovery.

Matheus de Oliveira sent in another revision of a patch to btree_gin to add
support for uuid, bool, name, bpchar and anyrange types.

Álvaro Herrera sent in another revision of a patch to allow partitioned tables
to have foreign keys that point to non-partitioned tables.

CharSyam sent in four revisions of a patch to add a sock check for Windows.

David Rowley sent in another revision of a patch to implement runtime partition
pruning.

Magnus Hagander sent in three more revisions of a patch to make it possible to
enable checksums online.

Michael Banck sent in another revision of a patch to verify checksums during
pgbasebackups.

Peter Geoghegan sent in another revision of a patch to add amcheck verification
of heap relations.

Pavel Stěhule sent in a patch to fix some breakage in PL/pgsql's plan cache.

Pavel Stěhule sent in a patch to fix tab completion in psql for the \sf and \ef.

Simon Riggs sent in a patch to ignore lazy vacuums in RunningTransactionData.

Dmitry Dolgov sent in a patch to rationalize the behavior of json[b]_to_tsvector
with respect to numeric values.

Yura Sokolov sent in a patch to make a hash table for xip in XidInMVCCSnapshot.