== PostgreSQL Weekly News - November 18, 2018 ==

== PostgreSQL Weekly News - November 18, 2018 ==

== PostgreSQL Product News ==

psqlODBC 11.00.0000 released.
https://odbc.postgresql.org/docs/release.html

pgCodeKeeper 5.2.0, an open-source database schema manager for PostgreSQL, released.
https://pgcodekeeper.org

== PostgreSQL Jobs for November ==

http://archives.postgresql.org/pgsql-jobs/2018-11/

== PostgreSQL Local ==

PGDay Argentina 2018 will be held in Buenos Aires on November 21, 2018.
http://www.pgday.com.ar/buenosaires2018/?lang=en

2Q PGConf will be on December 4-5, 2018 in Chicago, IL.
http://www.2qpgconf.com/

PGConf.ASIA 2018 will take place on December 10-12, 2018 in Akihabara, Tokyo,
Japan.
http://www.pgconf.asia/EN/2018/

FOSDEM PGDay 2019, a one day conference held before the main FOSDEM event will
be held in Brussels, Belgium, on Feb 1st, 2019.
https://2019.fosdempgday.org/

Prague PostgreSQL Developer Day 2019 (P2D2 2019) is a two-day
conference that will be held on February 13-14, 2019 in Prague, Czech Republic.
The CfP is open until January 4, 2018 at https://p2d2.cz/callforpapers
http://www.p2d2.cz/

PGConf India 2019 will be on February 13-15, 2019 in Bengaluru, Karnataka.
Proposals are due via https://goo.gl/forms/F9hRjOIsaNasVOAz2 by October 31st, 2017.
http://pgconf.in/

pgDay Paris 2019 will be held in Paris, France on March 12, 2019
at 199bis rue Saint-Martin. The CfP is open until November 30, 2018.
http://2019.pgday.paris/callforpapers/

PGConf APAC 2019 will be held in Singapore March 19-21, 2019. The CfP is open
at http://2019.pgconfapac.org/cfp through November 30, 2018.
http://2019.pgconfapac.org/

PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy. The CfP
is open at https://2019.pgday.it/en/blog/cfp and the Call for Workshops is at
https://2019.pgday.it/en/blog/cfw until January 15, 2019.
https://2019.pgday.it/en/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Andrew Dunstan pushed:

- Disable MSVC warning caused by recent snprintf.c changes. Discussion:
https://postgr.es/m/05f348de-0c79-d88d-69b7-434ef828bd4d@2ndQuadrant.com
https://git.postgresql.org/pg/commitdiff/7f284debaf1933b322ff54a643c6785362a034d3

- Silence MSVC warnings about redefinition of isnan. Some versions of perl.h
define isnan when the compiler is MSVC. To avoid warnings about this, undefine
the symbol before including perl.h and re-add the definition afterwards if it
wasn't recreated. Discussion:
https://postgr.es/m/caf0568e-3c1f-07fd-6914-d903f22560f2@2ndQuadrant.com
https://git.postgresql.org/pg/commitdiff/d5d7f7f3b78e00009ee87ad01d1fb87b28968f65

- fix typo.
https://git.postgresql.org/pg/commitdiff/79376e07128fbf3c92f4e1fb457be435afa2e6a6

Peter Eisentraut pushed:

- doc: Small run-time pruning doc fix. A note in ddl.sgml used to mention that
run-time pruning was only implemented for Append. When we got MergeAppend
support, this was updated to mention that MergeAppend is supported too. This
is slightly weird as it's not all that obvious what exactly isn't supported
when we mention: <para> Both of these behaviors are likely to be changed in a
future release of <productname>PostgreSQL</productname>. </para> This patch
updates this to mention that ModifyTable is unsupported, which makes the above
fragment make sense again. Author: David Rowley
<david(dot)rowley(at)2ndquadrant(dot)com>
https://git.postgresql.org/pg/commitdiff/253b3f6760e850945a91e2d872eab3b2802ffca2

- doc: Small punctuation improvement.
https://git.postgresql.org/pg/commitdiff/fc151211ef1f99934b8789591241525adfdee9d6

- pg_dump: Fix dumping of WITH OIDS tables. A table with OIDs that was the
first in the dump output would not get dumped with OIDs enabled. Fix that.
The reason was that the currWithOids flag was declared to be bool but actually
also takes a -1 value for "don't know yet". But under stdbool.h semantics,
that is coerced to true, so the required SET default_with_oids command is not
output again. Change the variable type to char to fix that. Reported-by:
Derek Nelson <derek(at)pipelinedb(dot)com>
https://git.postgresql.org/pg/commitdiff/6178f3cb798f2ea9011c55e99973f838719c550e

- doc: Fix minor whitespace issue. Reported-by: David G. Johnston
<david(dot)g(dot)johnston(at)gmail(dot)com>
https://git.postgresql.org/pg/commitdiff/d20dceaf50ea093234ef6a66bb2dae485468bea2

- Lower lock level for renaming indexes. Change lock level for renaming index
(either ALTER INDEX or implicitly via some other commands) from
AccessExclusiveLock to ShareUpdateExclusiveLock. One reason we need a strong
lock for relation renaming is that the name change causes a rebuild of the
relcache entry. Concurrent sessions that have the relation open might not be
able to handle the relcache entry changing underneath them. Therefore, we
need to lock the relation in a way that no one can have the relation open
concurrently. But for indexes, the relcache handles reloads specially in
RelationReloadIndexInfo() in a way that keeps changes in the relcache entry to
a minimum. As long as no one keeps pointers to rd_amcache and rd_options
around across possible relcache flushes, which is the case, this ought to be
safe. We also want to use a self-exclusive lock for correctness, so that
concurrent DDL doesn't overwrite the rename if they start updating while still
seeing the old version. Therefore, we use ShareUpdateExclusiveLock, which is
already used by other DDL commands that want to operate in a concurrent
manner. The reason this is interesting at all is that renaming an index is a
typical part of a concurrent reindexing workflow (CREATE INDEX CONCURRENTLY
new + DROP INDEX CONCURRENTLY old + rename back). And indeed a future
built-in REINDEX CONCURRENTLY might rely on the ability to do concurrent
renames as well. Reviewed-by: Andrey Klychkov <aaklychkov(at)mail(dot)ru>
Reviewed-by: Fabrízio de Royes Mello <fabriziomello(at)gmail(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/1531767486(dot)432607658(at)f357(dot)i(dot)mail(dot)ru
https://git.postgresql.org/pg/commitdiff/1b5d797cd4f7133ff0d18e123fcf41c67a5a7b0b

- Correct code comments for PartitionedRelPruneInfo struct. The comments above
the PartitionedRelPruneInfo struct incorrectly document how subplan_map and
subpart_map are indexed. This seems to have snuck in on 4e232364033. Author:
David Rowley <david(dot)rowley(at)2ndquadrant(dot)com>
https://git.postgresql.org/pg/commitdiff/52d8e899d0f0849f9dfc61a9f93ba857a8431216

- Update executor documentation for run-time partition pruning. With run-time
partition pruning, there is no longer necessarily an executor node for each
corresponding plan node. Author: David Rowley <david(dot)rowley(at)2ndquadrant(dot)com>
https://git.postgresql.org/pg/commitdiff/86a4819f691ba08bef0eba03dcf9be7bd1ca1c89

- A small tweak to some comments for PartitionKeyData. It was not really that
obvious that there's meant to be exactly 1 item in the partexprs List for each
zero-valued partattrs element. Some incorrect code using these fields was the
cause of CVE-2018-1052, so it's worthwhile to mention how they should be used
in the comments. Author: David Rowley <david(dot)rowley(at)2ndquadrant(dot)com>
https://git.postgresql.org/pg/commitdiff/7ac0069fb880b9b64223f104058c82773321851c

Michaël Paquier pushed:

- Fix incorrect author name in release notes. Author: Alexander Lakhin
Discussion: https://postgr.es/m/2f55f6d2-3fb0-d4f6-5c47-18da3a1117e0@gmail.com
https://git.postgresql.org/pg/commitdiff/ffb68980e3c9dc453812c70449ff2d36d5a7f6af

- Remove CommandCounterIncrement() after processing ON COMMIT DELETE. This
comes from f9b5b41, which is part of one the original commits that implemented
ON COMMIT actions. By looking at the truncation code, any CCI needed happens
locally when rebuilding indexes, so it looks safe to just remove this final
incrementation. Author: Michael Paquier Reviewed-by: Álvaro Herrera
Discussion: https://postgr.es/m/20181109024731.GF2652@paquier.xyz
https://git.postgresql.org/pg/commitdiff/52b70b1c7df5929465cf3dd8f4798e6f2e204f61

- Initialize TransactionState and user ID consistently at transaction start. If
a failure happens when a transaction is starting between the moment the
transaction status is changed from TRANS_DEFAULT to TRANS_START and the moment
the current user ID and security context flags are fetched via
GetUserIdAndSecContext(), or before initializing its basic fields, then those
may get reset to incorrect values when the transaction aborts, leaving the
session in an inconsistent state. One problem reported is that failing a
starting transaction at the first query of a session could cause several kinds
of system crashes on the follow-up queries. In order to solve that, move the
initialization of the transaction state fields and the call of
GetUserIdAndSecContext() in charge of fetching the current user ID close to
the point where the transaction status is switched to TRANS_START, where there
cannot be any error triggered in-between, per an idea of Tom Lane. This
properly ensures that the current user ID, the security context flags and that
the basic fields of TransactionState remain consistent even if the transaction
fails while starting. Reported-by: Richard Guo Diagnosed-By: Richard Guo
Author: Michael Paquier Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/CAN_9JTxECSb=pEPcb0a8d+6J+bDcOZ4=DgRo_B7Y5gRHJUM=Rw@mail.gmail.com
Backpatch-through: 9.4
https://git.postgresql.org/pg/commitdiff/b4721f39505b56dd7b556aef5428a0850230ca59

- Add flag values in WAL description to all heap records. Hexadecimal is
consistently used as format to not bloat too much the output but keep it
readable. This information is useful mainly for debugging purposes with for
example pg_waldump. Author: Michael Paquier Reviewed-by: Nathan Bossart,
Dmitry Dolgov, Andres Freund, Álvaro Herrera Discussion:
https://postgr.es/m/20180413034734.GE1552@paquier.xyz
https://git.postgresql.org/pg/commitdiff/3be97b97ed37b966173f027091f21d8a7605e2a5

Tom Lane pushed:

- Limit the number of index clauses considered in choose_bitmap_and().
classify_index_clause_usage() is O(N^2) in the number of distinct index qual
clauses it considers, because of its use of a simple search list to store
them. For nearly all queries, that's fine because only a few clauses will be
considered. But Alexander Kuzmenkov reported a machine-generated query with
80000 (!) index qual clauses, which caused this code to take forever.
Somewhat remarkably, this is the only O(N^2) behavior we now have for such a
query, so let's fix it. We can get rid of the O(N^2) runtime for cases like
this without much damage to the functionality of choose_bitmap_and() by
separating out paths with "too many" qual or pred clauses, and deeming them to
always be nonredundant with other paths. Then their clauses needn't go into
the search list, so it doesn't get too long, but we don't lose the ability to
consider bitmap AND plans altogether. I set the threshold for "too many" to
be 100 clauses per path, which should be plenty to ensure no change in
planning behavior for normal queries. There are other things we could do to
make this go faster, but it's not clear that it's worth any additional effort.
80000 qual clauses require a whole lot of work in many other places, too. The
code's been like this for a long time, so back-patch to all supported
branches. The troublesome query only works back to 9.5 (in 9.4 it fails with
stack overflow in the parser); so I'm not sure that fixing this in 9.4 has any
real-world benefit, but perhaps it does. Discussion:
https://postgr.es/m/90c5bdfa-d633-dabe-9889-3cf3e1acd443@postgrespro.ru
https://git.postgresql.org/pg/commitdiff/e3f005d974ce32ca9f74328067efafcb1217c87f

- Remove unused code in ECPG. scanner_init/scanner_finish weren't actually
called from anywhere, and the scanbuf variables they set up weren't used
either. Remove unused declaration for mm_realloc, too. John Naylor
Discussion:
https://postgr.es/m/CAJVSVGWGqY9YBs2EwtRUkbNv=hXkN8yRPOoD1wxE6COgvvrz5g@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/4766bcd9e292befba403619c8a57b3528ad3bd4b

- Align ECPG lexer more closely with the core and psql lexers. Make a bunch of
basically-cosmetic changes to reduce the diffs between the flex rules in
scan.l, psqlscan.l, and pgc.l. Reorder some code, adjust a lot of whitespace,
sync some comments, make use of flex start condition scopes to do that. There
are a few non-cosmetic changes in the ECPG lexer: * Bring over the decimalfail
rule (and support function process_integer_literal) so that ECPG will lex
"1..10" into the same tokens as the backend would. I'm not sure this makes
any visible difference to users, but I'm not sure it doesn't, either. *
<xdc><<EOF>> gets its own rule so as to produce a more on-point error message.
* Remove duplicate <SQL>{xdstart} rule. John Naylor, with a few additional
changes by me Discussion:
https://postgr.es/m/CAJVSVGWGqY9YBs2EwtRUkbNv=hXkN8yRPOoD1wxE6COgvvrz5g@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ec937d0805b205e5b33ed4f3cb54f40230a826e3

- Fix realfailN lexer rules to not make assumptions about input format. The
realfail1 and realfail2 backup-prevention rules always returned token type
FCONST, ignoring the possibility that what we've scanned is more appropriately
described as ICONST. I think that at the time that code was added, it might
actually have been safe to not distinguish; but since we started allowing
AS-less aliases in SELECT target lists, it's definitely legal to have a number
immediately followed by an identifier. In the SELECT case, it seems there's
no visible consequence because make_const() will change the type back to
integer anyway. But I'm worried that there are other contexts, or will be in
future, where it's more important to get the constant's type right. Hence,
use process_integer_literal to correctly determine which token type to return.
Arguably this is a bug fix, but given the lack of evidence of user-visible
problems, I'll refrain from back-patching. Discussion:
https://postgr.es/m/21364.1542136808@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/965a3d6be0702fa88c6eaa6946fa86218704c7d2

- Fix incorrect results for numeric data passed through an ECPG SQLDA. Numeric
values with leading zeroes were incorrectly copied into a SQLDA (SQL
Descriptor Area), leading to wrong results in ECPG programs. Report and patch
by Daisuke Higuchi. Back-patch to all supported versions. Discussion:
https://postgr.es/m/1803D792815FC24D871C00D17AE95905C71161@g01jpexmbkw24
https://git.postgresql.org/pg/commitdiff/ecfd5579561c3e720e696a42aadcd4b5b52aa581

- Second try at fixing numeric data passed through an ECPG SQLDA. In commit
ecfd55795, I removed sqlda.c's checks for ndigits != 0 on the grounds that we
should duplicate the state of the numeric value's digit buffer even when all
the digits are zeroes. However, that still isn't quite right, because another
possible state of the digit buffer is buf == digits == NULL (this occurs for a
NaN). As the code now stands, it'll invoke memcpy with a NULL source address
and zero bytecount, which we know a few platforms crash on. Hence, reinstate
the no-copy short-circuit, but make it test specifically for buf != NULL
rather than some other condition. In hindsight, the ndigits test (added by
commit f2ae9f9c3) was almost certainly meant to fix the NaN case not the
all-zeroes case as the associated thread alleged. As before, back-patch to
all supported versions. Discussion:
https://postgr.es/m/1803D792815FC24D871C00D17AE95905C71161@g01jpexmbkw24
https://git.postgresql.org/pg/commitdiff/06c723447b6b8fc7e394cd2ad23785a6e54e36da

- Add a timezone-specific variant of date_trunc(). date_trunc(field,
timestamptz, zone_name) performs truncation using the named time zone as
reference, rather than working in the session time zone as is the default
behavior. It's equivalent to date_trunc(field, timestamptz at time zone
zone_name) at time zone zone_name but it's faster, easier to type, and
arguably easier to understand. Vik Fearing and Tom Lane Discussion:
https://postgr.es/m/6249ffc4-2b22-4c1b-4e7d-7af84fedd7c6@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/600b04d6b5ef6c9ad3ea684aad40260bd60d5872

- Doc: remove claim that all \pset format options are unique in 1 letter. This
hasn't been correct since 9.3 added "latex-longtable". I left the phraseology
"Unique abbreviations are allowed" alone. It's correct as far as it goes, and
we are studiously refraining from specifying exactly what happens if you give
a non-unique abbreviation. (The answer in the back branches is "you get a
backwards-compatible choice", and the answer in HEAD will shortly be "you get
an error", but there seems no need to mention such details here.) Daniel
Vérité Discussion:
https://postgr.es/m/cb7e1caf-3ea6-450d-af28-f524903a030c@manitou-mail.org
https://git.postgresql.org/pg/commitdiff/51eaaafb850bc6d450cf2f63c0952078255ac4ca

- Make psql's "\pset format" command reject non-unique abbreviations. The
previous behavior of preferring the oldest match had the advantage of not
breaking existing scripts when we add a conflicting format name; but that
behavior was undocumented and fragile (it seems just luck that commit
add9182e5 didn't break it). Let's go over to the less mistake- prone approach
of complaining when there are multiple matches. Since this is a small
compatibility break, no back-patch. Daniel Vérité Discussion:
https://postgr.es/m/cb7e1caf-3ea6-450d-af28-f524903a030c@manitou-mail.org
https://git.postgresql.org/pg/commitdiff/eaf746a5b85a6a794e659f0954cf0015e42213f4

- Improve performance of partition pruning remapping a little.
ExecFindInitialMatchingSubPlans has to update the PartitionPruneState's
subplan mapping data to account for the removal of subplans it prunes.
However, that's only necessary if run-time pruning will also occur, so we can
skip it when that won't happen, which should result in not needing to do the
remapping in many cases. (We now need it only when some partitions are
potentially startup-time prunable and others are potentially run-time
prunable, which seems like an unusual case.) Also make some marginal
performance improvements in the remapping itself. These will mainly win if
most partitions got pruned by the startup-time pruning, which is perhaps a
debatable assumption in this context. Also fix some bogus comments, and
rearrange code to marginally reduce space consumption in the executor's
query-lifespan context. David Rowley, reviewed by Yoshikazu Imai Discussion:
https://postgr.es/m/CAKJS1f9+m6-di-zyy4B4AGn0y1B9F8UKDRigtBbNviXOkuyOpw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/34c9e455d0efcada8821ffaab61741c2e1153458

- Leave SIGTTIN/SIGTTOU signal handling alone in postmaster child processes.
For reasons lost in the mists of time, most postmaster child processes reset
SIGTTIN/SIGTTOU signal handling to SIG_DFL, with the major exception that
backend sessions do not. It seems like a pretty bad idea for any postmaster
children to do that: if stderr is connected to the terminal, and the user has
put the postmaster in background, any log output would result in the child
process freezing up. Hence, switch them all to doing what backends do, ie,
nothing. This allows them to inherit the postmaster's SIG_IGN setting. On
the other hand, manually-launched processes such as standalone backends will
have default processing, which seems fine. In passing, also remove useless
resets of SIGCONT and SIGWINCH signal processing. Perhaps the postmaster once
changed those to something besides SIG_DFL, but it doesn't now, so these are
just wasted (and confusing) syscalls. Basically, this propagates the changes
made in commit 8e2998d8a from backends to other postmaster children. Probably
the only reason these calls now exist elsewhere is that I missed changing
pgstat.c along with postgres.c at the time. Given the lack of field
complaints that can be traced to this, I don't presently feel a need to
back-patch. Discussion: https://postgr.es/m/5627.1542477392@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/125f551c8be101ec36ec5fed5befc8fbf7370e0a

- Avoid defining SIGTTIN/SIGTTOU on Windows. Setting them to SIG_IGN seems
unlikely to have any beneficial effect on that platform, and given the signal
numbering collision with SIGABRT, it could easily have bad effects. Given the
lack of field complaints that can be traced to this, I don't presently feel a
need to back-patch. Discussion:
https://postgr.es/m/5627.1542477392@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/37afc079abe1986b4af94aa8ec28cefd663aaae6

- Fix AC_REQUIRES breakage in LLVM autoconf tests. Any Autoconf macro that uses
AC_REQUIRES -- directly or indirectly -- must not be inside a plain shell "if"
test; if it is, whatever code gets pulled in by the AC_REQUIRES will also be
inside that "if". Instead of "if" we can use AS_IF, which knows how to get
this right (cf commit 01051a987). The only immediate problem from getting
this wrong was that AC_PROG_AWK had to be run twice, once inside the "if llvm"
block and once in the main line. However, it broke a different patch I'm
about to submit more thoroughly.
https://git.postgresql.org/pg/commitdiff/0b59b0e8bcc9e85bad6fffa7828344db1a36f20a

Thomas Munro pushed:

- Fix possible buffer overrun in hba.c. Coverty reports a possible buffer
overrun in the code that populates the pg_hba_file_rules view. It may not be
a live bug due to restrictions on options that can be used together, but let's
increase MAX_HBA_OPTIONS and correct a nearby misleading comment. Back-patch
to 10 where this code arrived. Reported-by: Julian Hsiao Discussion:
https://postgr.es/m/CADnGQpzbkWdKS2YHNifwAvX5VEsJ5gW49U4o-7UL5pzyTv4vTg%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/6a3dcd28568a04b6e4aea2bf41ea2c7e9c7b0e96

- Fix const correctness warning. Per buildfarm.
https://git.postgresql.org/pg/commitdiff/b59d4d6c36e14ba4097fa8dac3d31ca86f7f507d

- Fix handling of HBA ldapserver with multiple hostnames. Commit 35c0754f
failed to handle space-separated lists of alternative hostnames in ldapserver,
when building a URI for ldap_initialize() (OpenLDAP). Such lists need to be
expanded to space-separated URIs. Repair. Back-patch to 11, to fix bug
report #15495. Author: Thomas Munro Reported-by: Renaud Navarro Discussion:
https://postgr.es/m/15495-2c39fc196c95cd72%40postgresql.org
https://git.postgresql.org/pg/commitdiff/257ef3cd4fec7ca1213f31b660486b492b1c7031

- Use 64 bit type for BufFileSize(). BufFileSize() can't use off_t, because
it's only 32 bits wide on some systems. BufFile objects can have many 1GB
segments so the total size can exceed 2^31. The only known client of the
function is parallel CREATE INDEX, which was reported to fail when building
large indexes on Windows. Though this is technically an ABI break on
platforms with a 32 bit off_t and we might normally avoid back-patching it,
the function is brand new and thus unlikely to have been discovered by
extension authors yet, and it's fairly thoroughly broken on those platforms
anyway, so just fix it. Defect in 9da0cc35. Bug #15460. Back-patch to 11,
where this function landed. Author: Thomas Munro Reported-by: Paul van der
Linden, Pavel Oskin Reviewed-by: Peter Geoghegan Discussion:
https://postgr.es/m/15460-b6db80de822fa0ad%40postgresql.org Discussion:
https://postgr.es/m/CAHDGBJP_GsESbTt4P3FZA8kMUKuYxjg57XHF7NRBoKnR%3DCAR-g%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/aa5518304213a762b62ebe3dbb057d220d8976eb

- Increase the number of possible random seeds per time period. Commit 197e4af9
refactored the initialization of the libc random() seed, but reduced the
number of possible seeds values that could be chosen in a given time period.
This negation of the effects of commit 98c50656c was unintentional. Replace
with code that shifts the fast moving timestamp bits left, similar to the
original algorithm (though not the previous float-tolerating coding, which is
no longer necessary). Author: Thomas Munro Reported-by: Noah Misch
Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/20181112083358.GA1073796%40rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/5b0ce3ec334bb65bbab4aba78f204f986c356e80

- Further adjustment to random() seed initialization. Per complaint from Tom
Lane, don't chomp the timestamp at 32 bits, so we can shift in some of its
higher bits. Discussion: https://postgr.es/m/14712.1542253115%40sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/ab8984f52d9b99234d23e6fb7b73cf4c12b3ac14

Amit Kapila pushed:

- Fix the initialization of atomic variables introduced by the. group clearing
mechanism. Commits 0e141c0fbb and baaf272ac9 introduced initialization of
atomic variables in InitProcess which means that it's not safe to look at
those for backends that aren't currently in use. Fix that by initializing
them during postmaster startup. Reported-by: Andres Freund Author: Amit
Kapila Backpatch-through: 9.6 Discussion:
https://postgr.es/m/20181027104138.qmbbelopvy7cw2qv@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/a53bc135fb07d600869917661d65f45506165c00

- Fix the omission in docs. Commit 5373bc2a08 has added type for background
workers but forgot to update at one place in the documentation. Reported-by:
John Naylor Author: John Naylor Reviewed-by: Amit Kapila Backpatch-through: 11
Discussion:
https://postgr.es/m/CAJVSVGVmvgJ8Lq4WBxC3zV5wf0txdCqRSgkWVP+jaBF=HgWscA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/621a8ac5af8cf7c7799e8b704108b83be89267dd

Álvaro Herrera pushed:

- Add INSERT ON CONFLICT test on partitioned tables with transition table. This
case was uncovered by existing tests, so breakage went undetected. Make sure
it remains stable. Extracted from a larger patch by Author: David Rowley
Reviewed-by: Amit Langote Discussion:
https://postgr.es/m/CAKJS1f-aGCJ5H7_hiSs5PhWs6Obmj+vGARjGymqH1=o5PcrNnQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/9079fe60b2142da6e89403b9a9504e75124c085d

- geo_ops.c: Clarify comments and function arguments. These functions were not
crystal clear about what their respective APIs are. Make an effort to improve
that. Emre's patch was correct AFAICT, but I (Álvaro) felt the need to
improve a few comments a bit more. Any resulting errors are my own. Per
complaint from Coverity, Ning Yu, and Tom Lane. Author: Emre Hasegeli, Álvaro
Herrera Reviewed-by: Tomas Vondra, Álvaro Herrera Discussion:
https://postgr.es/m/26769.1533090136@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/74514bd4a58dcf763d105e60a41a5f1b3eea6865

- pgbench: introduce a RandomState struct. This becomes useful when used to
retry a transaction after a serialization error or deadlock abort. (We don't
yet have that feature, but this is preparation for it.) While at it, use
separate random state for thread administratrivia such as deciding which
script to run, how long to delay for throttling, or whether to log a message
when sampling; this not only makes these tasks independent of each other, but
makes the actual thread run deterministic. Author: Marina Polyakova
Reviewed-by: Fabien Coelho Discussion:
https://postgr.es/m/72a0d590d6ba06f242d75c2e641820ec@postgrespro.ru
https://git.postgresql.org/pg/commitdiff/409231919443984635b7ae9b7e2e261ab984eb1e

- Avoid re-typedef'ing PartitionTupleRouting. Apparently, gcc on macOS (?)
doesn't like it. Per buildfarm.
https://git.postgresql.org/pg/commitdiff/0201d79a5549e3375ea5e5aee351378399453f15

- Redesign initialization of partition routing structures. This speeds up write
operations (INSERT, UPDATE, DELETE, COPY, as well as the future MERGE) on
partitioned tables. This changes the setup for tuple routing so that it does
far less work during the initial setup and pushes more work out to when
partitions receive tuples. PartitionDispatchData structs for sub-partitioned
tables are only created when a tuple gets routed through it. The possibly
large arrays in the PartitionTupleRouting struct have largely been removed.
The partitions[] array remains but now never contains any NULL gaps.
Previously the NULLs had to be skipped during ExecCleanupTupleRouting(), which
could add a large overhead to the cleanup when the number of partitions was
large. The partitions[] array is allocated small to start with and only
enlarged when we route tuples to enough partitions that it runs out of space.
This allows us to keep simple single-row partition INSERTs running quickly.
Redesign The arrays in PartitionTupleRouting which stored the tuple
translation maps have now been removed. These have been moved out into a
PartitionRoutingInfo struct which is an additional field in ResultRelInfo.
The find_all_inheritors() call still remains by far the slowest part of
ExecSetupPartitionTupleRouting(). This commit just removes the other slow
parts. In passing also rename the tuple translation maps from being
ParentToChild and ChildToParent to being RootToPartition and PartitionToRoot.
The old names mislead you into thinking that a partition of some
sub-partitioned table would translate to the rowtype of the sub-partitioned
table rather than the root partitioned table. Authors: David Rowley and Amit
Langote, heavily revised by Álvaro Herrera Testing help from Jesper Pedersen
and Kato Sho. Discussion:
https://postgr.es/m/CAKJS1f_1RJyFquuCKRFHTdcXqoPX-PYqAd7nz=GVBwvGh4a6xA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3f2393edefa5ef2b6970a5a2fa2c7e9c55cc10cf

Andres Freund pushed:

- Make reformat-dat-files, reformat-dat-files VPATH safe. The
reformat_dat_file.pl script, added by 372728b0d49552641, supported all the
necessary options to make it work in a VPATH build, but the makefile
invocations didn't take VPATH into account. Fix that. Discussion:
https://postgr.es/m/20181115185303.d2z7wonx23mdfvd3@alap3.anarazel.de
Backpatch: 11-, where 372728b0d49552641 was merged
https://git.postgresql.org/pg/commitdiff/6a744413ea0c0c114348c4c010aa9f909c83d3f3

- Rationalize expression context reset in ExecModifyTable(). The current
pattern of reseting expressions both in ExecProcessReturning() and
ExecOnConflictUpdate() makes it harder than necessary to reason about memory
lifetimes. It also requires materializing slots unnecessarily, although this
patch doesn't take advantage of the fact that that's not necessary anymore.
Instead reset the expression context once for each input tuple. Author:
Ashutosh Bapat Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/c058fc2a2b8ea25015b211186c88a6a3006f5937

- Rejigger materializing and fetching a HeapTuple from a slot. Previously
materializing a slot always returned a HeapTuple. As current work aims to
reduce the reliance on HeapTuples (so other storage systems can work
efficiently), that needs to change. Thus split the tasks of materializing a
slot (i.e. making it independent from the underlying storage / other memory
contexts) from fetching a HeapTuple from the slot. For brevity, allow to
fetch a HeapTuple from a slot and materializing the slot at the same time,
controlled by a parameter. For now some callers of ExecFetchSlotHeapTuple,
with materialize = true, expect that changes to the heap tuple will be
reflected in the underlying slot. Those places will be adapted in due course,
so while not pretty, that's OK for now. Also rename ExecFetchSlotTuple to
ExecFetchSlotHeapTupleDatum and ExecFetchSlotTupleDatum to
ExecFetchSlotHeapTupleDatum, as it's likely that future storage methods will
need similar methods. There already is ExecFetchSlotMinimalTuple, so the new
names make the naming scheme more coherent. Author: Ashutosh Bapat and Andres
Freund, with changes by Amit Khandekar Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/763f2edd92095b1ca2f4476da073a28505c13820

- Don't generate tuple deforming functions for virtual slots. Virtual tuple
table slots never need tuple deforming. Therefore, if we know at expression
compilation time, that a certain slot will always be virtual, there's no need
to create a tuple deforming routine for it. Author: Andres Freund Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/7ef04e4d2cb287e4e28b87f24b4b36ef4e07530b

- Verify that expected slot types match returned slot types. This is important
so JIT compilation knows what kind of tuple slot the deforming routine can
expect. There's also optimization potential for expression initialization
without JIT compilation. It e.g. seems plausible to elide EEOP_*_FETCHSOME ops
entirely when dealing with virtual slots. Author: Andres Freund Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/15d8f83128e15de97de61430d0b9569f5ebecc26

- Compute information about EEOP_*_FETCHSOME at expression init time.
Previously this information was computed when JIT compiling an expression.
But the information is useful for assertions in the non-JIT case too (for
assertions), therefore it makes sense to move it. This will, in a followup
commit, allow to treat different slot types differently. E.g. for virtual
slots there's no need to generate a JIT function to deform the slot. Author:
Andres Freund Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/675af5c01e297262cd20d7416f7e568393c22c6e

- Introduce notion of different types of slots (without implementing them).
Upcoming work intends to allow pluggable ways to introduce new ways of storing
table data. Accessing those table access methods from the executor requires
TupleTableSlots to be carry tuples in the native format of such storage
methods; otherwise there'll be a significant conversion overhead. Different
access methods will require different data to store tuples efficiently (just
like virtual, minimal, heap already require fields in TupleTableSlot). To
allow that without requiring additional pointer indirections, we want to have
different structs (embedding TupleTableSlot) for different types of slots.
Thus different types of slots are needed, which requires adapting creators of
slots. The slot that most efficiently can represent a type of tuple in an
executor node will often depend on the type of slot a child node uses.
Therefore we need to track the type of slot is returned by nodes, so parent
slots can create slots based on that. Relatedly, JIT compilation of tuple
deforming needs to know which type of slot a certain expression refers to, so
it can create an appropriate deforming function for the type of tuple in the
slot. But not all nodes will only return one type of slot, e.g. an append
node will potentially return different types of slots for each of its
subplans. Therefore add function that allows to query the type of a node's
result slot, and whether it'll always be the same type (whether it's fixed).
This can be queried using ExecGetResultSlotOps(). The scan, result, inner,
outer type of slots are automatically inferred from ExecInitScanTupleSlot(),
ExecInitResultSlot(), left/right subtrees respectively. If that's not correct
for a node, that can be overwritten using new fields in PlanState. This
commit does not introduce the actually abstracted implementation of different
kind of TupleTableSlots, that will be left for a followup commit. The
different types of slots introduced will, for now, still use the same backing
implementation. While this already partially invalidates the big comment in
tuptable.h, it seems to make more sense to update it later, when the different
TupleTableSlot implementations actually exist. Author: Ashutosh Bapat and
Andres Freund, with changes by Amit Khandekar Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/1a0586de3657cd35581f0639c87d5050c6197bb7

- Add dummy field to currently empty struct TupleTableSlotOps. Per MSVC
complaint on buildfarm member dory.
https://git.postgresql.org/pg/commitdiff/f92cd7392386147f6a16787c1d5c78d0e9b7cf34

- Fix slot type assumptions for nodeGather[Merge]. The assumption made in
1a0586de3657c was wrong, as evidenced by buildfarm failure on locust, which
runs with force_parallel_mode=regress. The tuples accessed in either nodes
are in the outer slot, and we can't trivially rely on the slot type being
known because the leader might execute the subsidiary node directly, or via
the tuple queue on a worker. In the latter case the tuple will always be a
heaptuple slot, but in the former, it'll be whatever the subsidiary node
returns.
https://git.postgresql.org/pg/commitdiff/a387a3dff9001225ad571ff2755d139f5bd193b3

- Inline hot path of slot_getsomeattrs(). This yields a minor speedup, which
roughly balances the loss from the upcoming introduction of callbacks to do
some operations on slots. Author: Andres Freund Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/a7aa608e0f5910f9c73a530a66142c08e3d9043a

- Make TupleTableSlots extensible, finish split of existing slot type. This
commit completes the work prepared in 1a0586de36, splitting the old
TupleTableSlot implementation (which could store buffer, heap, minimal and
virtual slots) into four different slot types. As described in the
aforementioned commit, this is done with the goal of making tuple table slots
extensible, to allow for pluggable table access methods. To achieve runtime
extensibility for TupleTableSlots, operations on slots that can differ between
types of slots are performed using the TupleTableSlotOps struct provided at
slot creation time. That includes information from the size of TupleTableSlot
struct to be allocated, initialization, deforming etc. See the struct's
definition for more detailed information about callbacks TupleTableSlotOps. I
decided to rename TTSOpsBufferTuple to TTSOpsBufferHeapTuple and
ExecCopySlotTuple to ExecCopySlotHeapTuple, as that seems more consistent with
other naming introduced in recent patches. There's plenty optimization
potential in the slot implementation, but according to benchmarking the state
after this commit has similar performance characteristics to before this set
of changes, which seems sufficient. There's a few changes in
execReplication.c that currently need to poke through the slot abstraction,
that'll be repaired once the pluggable storage patchset provides the necessary
infrastructure. Author: Andres Freund and Ashutosh Bapat, with changes by
Amit Khandekar Discussion:
https://postgr.es/m/20181105210039.hh4vvi4vwoq5ba2q@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/4da597edf1bae0cf0453b5ed6fc4347b6334dfe1

- Fix some spurious new compiler warnings in MSVC. Per buildfarm animal
bowerbird. Discussion:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bowerbird&dt=2018-11-17%2002%3A30%3A20
https://git.postgresql.org/pg/commitdiff/73616126b4dcd01c833b06d4572959734f38b163

Tomáš Vondra:

- Add valgrind suppressions for wcsrtombs optimizations. wcsrtombs (called
through wchar2char from common functions like lower, upper, etc.) uses various
optimizations that may look like access to uninitialized data, triggering
valgrind reports. For example AVX2 instructions load data in 256-bit chunks,
and gconv does something similar with 32-bit chunks. This is faster than
accessing the bytes one by one, and the uninitialized part of the buffer is
not actually used. So suppress the bogus reports. The exact stack depends on
possible optimizations - it might be AVX, SSE (as in the report by Aleksander
Alekseev) or something else. Hence the last frame is wildcarded, to deal with
this. Backpatch all the way back to 9.4. Author: Tomas Vondra Discussion:
https://www.postgresql.org/message-id/flat/90ac0452-e907-e7a4-b3c8-15bd33780e62%402ndquadrant.com
Discussion:
https://www.postgresql.org/message-id/20180220150838.GD18315@e733.localdomain
https://git.postgresql.org/pg/commitdiff/d3bbc4b96a5b4d055cf636596c6865913a099929

== Pending Patches ==

Ryo Matsumura sent in another revision of a patch to add a bytea type to ECPG.

Michaël Paquier sent in another revision of a patch to add a TAP test checking
data consistency on standby with minRecoveryPoint.

Edmund Horner sent in another revision of a patch to improve how TID scans work.

Thomas Munro sent in a patch to refactor the checkpointer's data sync request
queue.

Kyotaro HORIGUCHI sent in another revision of a patch to change the way the stats
collector stores its information from files to shared memory, so no more
pg_stats_tmp.

Peter Geoghegan sent in another revision of a patch to make nbtree indexes have
unique keys in tuples.

Pavel Stěhule sent in a patch to add a unique statement ID to PL/pgsql
statements.

Sergei Kornilov sent in another revision of a patch to refactor the recovery
API.

Christoph Berg sent in a patch to pass COPT and PROFILE to CXXFLAGS as well.

Thomas Munro sent in a patch to add a quick hack to try wrapping a DSA area in a
MemoryContext.

Haribabu Kommi and Amit Kapila traded patches to extend pg_stat_statements_reset
to reset statistics specific to a particular user/db/query.

Etsuro Fujita and Tom Lane traded patches to fix some corner case handling of
SIGPIPE.

Kyotaro HORIGUCHI sent in another revision of a patch to implement WAL logging
skip.

Evgeniy Efimkin sent in a patch to add a special role for CREATE SUBSCRIPTION.

Raúl Marín Rodríguez sent in a patch to pg_config to avoid leaking configdata.

James Coleman sent in another revision of a patch to prove IS NOT NULL inference
for large arrays.

Amit Langote sent in two more revisions of a patch to speed up planning with
partitions.

Andres Freund sent in two revisions of a patch to remove WITH OIDs support.

Robert Haas and Amit Langote traded patches to lay the infrastructure for the
upcoming {AT,DE}TACH PARTITION CONCURRENTLY.

Dilip Kumar sent in four more revisions of a patch to implement UNDO logs.

Dmitry Dolgov sent in two more revisions of a patch to implement index skip
scans.

Masahiko Sawada sent in another revision of a patch to support atomic commits
among multiple foreign servers and use that infrastructure in the PostgreSQL
FDW.

Peter Geoghegan sent in another revision of a patch to add a pg_depend index
scan tiebreaker column.

Nikita Glukhov sent in another revision of a patch to add opclass parameters.

Thomas Munro sent in a PoC patch to cache file sizes to avoid lseek() calls.

Haribabu Kommi sent in another revision of a patch to enhance and document
pluggable storage.

Thomas Munro sent in another revision of a patch to add DNS SRV support for LDAP
server discovery.

John Naylor sent in a patch to reformat the .dat files to support the removal of
WITH OIDS tables.

David Fetter sent in a patch to document the fact that the node a PUBLICATION
is on cannot be changed.

Melanie Plageman sent in a patch to refactor semijoin selectivity estimation.

Justin Pryzby sent in another revision of a patch to fix some comments in
src/backend/jit/llvm/llvmjit_deform.c.

Thomas Munro sent in another revision of a patch to add a WL_EXIT_ON_PM_DEATH
pseudo-event.

Fabien COELHO sent in another revision of a patch to pgbench to clean up
doCustom.

Álvaro Herrera sent in another revision of a patch to fix COPY FREEZE for
partitioned tables.

Fabien COELHO sent in another revision of a patch to fix psql's \conninfo and
\connect when using hostaddr.

Fabien COELHO sent in two more revisions of a patch to pgbench to enable it to
store select results into variables.

Álvaro Herrera sent in another revision of a patch to add access method
information to psql via \dA and friends.

Thomas Munro sent in another revision of a patch to fix some issues around fsync
requests which misbehave on certain types of failure.

Alexander Lakhin sent in another revision of a patch to fix some corner cases in
`make installcheck`.

Shay Rojansky sent in a patch to allow UNLISTEN during recovery.

Fabien COELHO sent in another revision of a patch to improve the descriptions of
default privileges.

Tomáš Vondra sent in a patch to fix some infelicities in the interaction between
logical decoding and VACUUM FULL / CLUSTER.

Tom Lane sent in a patch to fix AC_CHECK_DECLS to do the right thing with clang.