== PostgreSQL Weekly News - March 31, 2019 ==

== PostgreSQL Weekly News - March 31, 2019 ==

== PostgreSQL Product News ==

Pgpool-II 4.0.4, 3.7.9, 3.6.16, 3.5.20 and 3.4.23 released.
http://pgpool.net/mediawiki/index.php/Downloads

padnag 1.2.4, a tool for synchronizing PostgreSQL roles with
Active Directory users and groups, released.
https://padnag.io/

pg_activity 1.5, a top-like application for PostgreSQL server activity
monitoring, released.
https://github.com/julmon/pg_activity/releases/tag/v1.5.0

== PostgreSQL Jobs for March ==

http://archives.postgresql.org/pgsql-jobs/2019-03/

== PostgreSQL Local ==

The German-speaking PostgreSQL Conference 2019 will take place on May 10, 2019
in Leipzig.
http://2019.pgconf.de/

PGDay.IT 2019 will take place May 16th and May 17th in Bologna, Italy.
https://2019.pgday.it/en/

PGCon 2019 will take place in Ottawa on May 28-31, 2019.
https://www.pgcon.org/2019

Swiss PGDay 2019 will take place in Rapperswil (near Zurich) on June 28, 2019.
The CfP is open through April 18, 2019, and registration is open.
http://www.pgday.ch/2019/

PostgresLondon 2019 will be July 2-3, 2019 with an optional training day on
July 1.
http://postgreslondon.org

PGConf.Brazil 2019 is on August 1-3 2019 in São Paulo.
http://pgconf.com.br

The first Austrian pgDay, will take place September 6, 2019 at the Hilton Garden
Inn in Wiener Neustadt. The CfP is open until April 1, 2019.
https://pgday.at/en/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david(at)fetter(dot)org(dot)

== Applied Patches ==

Tom Lane pushed:

- Avoid double-free in vacuumlo error path. The code would do "PQclear(res)"
twice if lo_unlink failed, evidently due to careless thinking about how far
out a "break" would break. Remove the extra PQclear and adjust the loop logic
so that we'll fall out of both levels of loop after an error, as was clearly
the intent. Spotted by Coverity. I have no idea why it took this long to
notice, since the bug has been there since commit 67ccbb080. Accordingly,
back-patch to all supported branches.
https://git.postgresql.org/pg/commitdiff/bd9396a0b2d42497f1196af74b327e00cb6af435

- Sort dependent objects before reporting them in DROP ROLE. Commit 8aa9dd74b
didn't quite finish the job in this area after all, because DROP ROLE has a
code path distinct from DROP OWNED BY, and it was still reporting dependent
objects in whatever order the index scan returned them in. Buildfarm
experience shows that index ordering of equal-keyed objects is significantly
less stable than before in the wake of using heap TIDs as tie-breakers. So if
we try to hide the unstable ordering by suppressing DETAIL reports, we're just
going to end up having to do that for every DROP that reports multiple
objects. That's not great from a coverage or problem-detection standpoint,
and it's something we'll inevitably forget in future patches, leading to more
iterations of fixing-an- unstable-result. So let's just bite the bullet and
sort here too. Discussion:
https://postgr.es/m/E1h6eep-0001Mw-Vd@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/af6550d34466b3093edda54a0cc5a6f220d321b7

- Un-hide most cascaded-drop details in regression test results. Now that the
ordering of DROP messages ought to be stable everywhere, we should not need
these kluges of hiding DETAIL output just to avoid unstable ordering. Hiding
it's not great for test coverage, so let's undo that where possible. In a
small number of places, it's necessary to leave it in, for example because the
output might include a variable pg_temp_nnn schema name. I also left things
alone in places where the details would depend on other regression test
scripts, e.g. plpython_drop.sql. Perhaps buildfarm experience will show this
to be a bad idea, but if so I'd like to know why. Discussion:
https://postgr.es/m/E1h6eep-0001Mw-Vd@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/940311e4bb32a5fe99155052e41179c88b5d48af

- Further code review for new integerset code. Mostly cosmetic adjustments, but
I added a more reliable method of detecting whether an iteration is in
progress.
https://git.postgresql.org/pg/commitdiff/f7ff0ae8428f9001eab244cb2535f40d29f218e9

- Suppress Append and MergeAppend plan nodes that have a single child. If
there's only one child relation, the Append or MergeAppend isn't doing
anything useful, and can be elided. It does have a purpose during planning
though, which is to serve as a buffer between parent and child Var numbering.
Therefore we keep it all the way through to setrefs.c, and get rid of it only
after fixing references in the plan level(s) above it. This works largely the
same as setrefs.c's ancient hack to get rid of no-op SubqueryScan nodes, and
can even share some code with that. Note the change to make setrefs.c use
apply_tlist_labeling rather than ad-hoc code. This has the effect of
propagating the child's resjunk and ressortgroupref labels, which formerly
weren't propagated when removing a SubqueryScan. Doing that is demonstrably
necessary for the [Merge]Append cases, and seems harmless for SubqueryScan, if
only because trivial_subqueryscan is afraid to collapse cases where the
resjunk marking differs. (I suspect that restriction could now be removed,
though it's unclear that it'd make any new matches possible, since the outer
query can't have references to a child resjunk column.) David Rowley,
reviewed by Alvaro Herrera and Tomas Vondra Discussion:
https://postgr.es/m/CAKJS1f_7u8ATyJ1JGTMHFoKDvZdeF-iEBhs+sM_SXowOr9cArg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/8edd0e79460b414b1d971895312e549e95e12e4f

- Doc: clarify that REASSIGN OWNED doesn't handle default privileges. It doesn't
touch regular privileges either, but only the latter was explicitly stated.
Discussion:
https://postgr.es/m/155348282848.9808.12629518043813943231@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/b3bd819632701d53580d9433f3f80c529865f86d

- Improve planner's selectivity estimates for inequalities on CTID. We were
getting just DEFAULT_INEQ_SEL for comparisons such as "ctid >= constant", but
it's possible to do a lot better if we don't mind some assumptions about the
table's tuple density being reasonably uniform. There are already assumptions
much like that elsewhere in the planner, so that hardly seems like much of an
objection. Extracted from a patch set that also proposes to introduce a
special executor node type for such queries. Not sure if that's going to make
it into v12, but improving the selectivity estimate is useful independently of
that. Edmund Horner, reviewed by David Rowley Discussion:
https://postgr.es/m/CAMyN-kB-nFTkF=VA_JPwFNo08S0d-Yk0F741S2B7LDmYAi8eyA@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/f7111f72d2fd31979c6d473cdde56ba8e7aba26a

- Get rid of duplicate child RTE for a partitioned table. We've been creating
duplicate RTEs for partitioned tables just because we do so for regular
inheritance parent tables. But unlike regular-inheritance parents which are
themselves regular tables and thus need to be scanned, partitioned tables
don't need the extra RTE. This makes the conditions for building a child RTE
the same as those for building an AppendRelInfo, allowing minor simplification
in expand_single_inheritance_child. Since the planner's actual processing is
driven off the AppendRelInfo list, nothing much changes beyond that, we just
have one fewer useless RTE entry. Amit Langote, reviewed and hacked a bit by
me Discussion:
https://postgr.es/m/9d7c5112-cb99-6a47-d3be-cf1ee6862a1d@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/e8d5dd6be79ed532710922105c31a2d25b14430f

- Fix oversight in data-type change for autovacuum_vacuum_cost_delay. Commit
caf626b2c missed that the relevant reloptions entry needs to be moved from the
intRelOpts[] array to realRelOpts[]. Somewhat surprisingly, it seems to work
anyway, perhaps because the desired default and limit values are all integers.
We ought to have either a simpler data structure or better cross-checking
here, but that's for another patch. Nikolay Shaplov Discussion:
https://postgr.es/m/4861742.12LTaSB3sv@x200m
https://git.postgresql.org/pg/commitdiff/7c366ac969ce060c4080fbdfe74a0131ae4bf6bb

- Add ORDER BY to more ICU regression test cases. Commit c77e12208 didn't fully
fix the problem. Per buildfarm and local testing.
https://git.postgresql.org/pg/commitdiff/8994cc6ffc8828569a39139996a0b0b8348ca036

- Build "other rels" of appendrel baserels in a separate step. Up to now,
otherrel RelOptInfos were built at the same time as baserel RelOptInfos,
thanks to recursion in build_simple_rel(). However, nothing in
query_planner's preprocessing cares at all about otherrels, only baserels, so
we don't really need to build them until just before we enter make_one_rel.
This has two benefits: * create_lateral_join_info did a lot of extra work to
propagate lateral-reference information from parents to the correct children.
But if we delay creation of the children till after that, it's trivial (and
much harder to break, too). * Since we have all the restriction quals
correctly assigned to parent appendrels by this point, it'll be possible to do
plan-time pruning and never make child RelOptInfos at all for partitions that
can be pruned away. That's not done here, but will be later on. Amit
Langote, reviewed at various times by Dilip Kumar, Jesper Pedersen, Yoshikazu
Imai, and David Rowley Discussion:
https://postgr.es/m/9d7c5112-cb99-6a47-d3be-cf1ee6862a1d@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/53bcf5e3dbeaed5baf9d09b124cf196d247c54ea

- Suppress uninitialized-variable warning. Apparently Andres' compiler is smart
enough to see that hpage must be initialized before use ... but mine isn't.
https://git.postgresql.org/pg/commitdiff/a51cc7e9e60701d31032669011adcda8b4621d93

- Avoid passing query tlist around separately from root->processed_tlist. In the
dim past, the planner kept the fully-processed version of the query targetlist
(the result of preprocess_targetlist) in grouping_planner's local variable
"tlist", and only grudgingly passed it to individual other routines as needed.
Later we discovered a need to still have it available after grouping_planner
finishes, and invented the root->processed_tlist field for that purpose, but
it wasn't used internally to grouping_planner; the tlist was still being
passed around separately in the same places as before. Now comes a proposed
patch to allow appendrel expansion to add entries to the processed tlist, well
after preprocess_targetlist has finished its work. To avoid having to pass
around the tlist explicitly, it's proposed to allow appendrel expansion to
modify root->processed_tlist. That makes aliasing the tlist with assorted
parameters and local variables really scary. It would accidentally work as
long as the tlist is initially nonempty, because then the List header won't
move around, but it's not exactly hard to think of ways for that to break.
Aliased values are poor programming practice anyway. Hence, get rid of local
variables and parameters that can be identified with root->processed_tlist, in
favor of just using that field directly. And adjust comments to match. (Some
of the new comments speak as though it's already possible for appendrel
expansion to modify the tlist; that's not true yet, but will happen in a later
patch.) Discussion:
https://postgr.es/m/9d7c5112-cb99-6a47-d3be-cf1ee6862a1d@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/333ed246c6f351c4e8fe22c764b97793c4101b00

- Avoid crash in partitionwise join planning under GEQO. While trying to plan a
partitionwise join, we may be faced with cases where one or both input
partitions for a particular segment of the join have been pruned away. In
HEAD and v11, this is problematic because earlier processing didn't bother to
make a pruned RelOptInfo fully valid. With an upcoming patch to make
partition pruning more efficient, this'll be even more problematic because
said RelOptInfo won't exist at all. The existing code attempts to deal with
this by retroactively making the RelOptInfo fully valid, but that causes
crashes under GEQO because join planning is done in a short-lived memory
context. In v11 we could probably have fixed this by switching to the
planner's main context while fixing up the RelOptInfo, but that idea doesn't
scale well to the upcoming patch. It would be better not to mess with the
base-relation data structures during join planning, anyway --- that's just a
recipe for order-of-operations bugs. In many cases, though, we don't actually
need the child RelOptInfo, because if the input is certainly empty then the
join segment's result is certainly empty, so we can skip making a join plan
altogether. (The existing code ultimately arrives at the same conclusion, but
only after doing a lot more work.) This approach works except when the
pruned-away partition is on the nullable side of a LEFT, ANTI, or FULL join,
and the other side isn't pruned. But in those cases the existing code leaves
a lot to be desired anyway --- the correct output is just the result of the
unpruned side of the join, but we were emitting a useless outer join against a
dummy Result. Pending somebody writing code to handle that more nicely, let's
just abandon the partitionwise-join optimization in such cases. When the
modified code skips making a join plan, it doesn't make a join RelOptInfo
either; this requires some upper-level code to cope with nulls in part_rels[]
arrays. We would have had to have that anyway after the upcoming patch.
Back-patch to v11 since the crash is demonstrable there. Discussion:
https://postgr.es/m/8305.1553884377@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/7ad6498fd5a654de6e743814c36cf619a3b5ddb6

- Speed up planning when partitions can be pruned at plan time. Previously, the
planner created RangeTblEntry and RelOptInfo structs for every partition of a
partitioned table, even though many of them might later be deemed
uninteresting thanks to partition pruning logic. This incurred significant
overhead when there are many partitions. Arrange to postpone creation of these
data structures until after we've processed the query enough to identify
restriction quals for the partitioned table, and then apply partition pruning
before not after creation of each partition's data structures. In this way we
need not open the partition relations at all for partitions that the planner
has no real interest in. For queries that can be proven at plan time to
access only a small number of partitions, this patch improves the practical
maximum number of partitions from under 100 to perhaps a few thousand. Amit
Langote, reviewed at various times by Dilip Kumar, Jesper Pedersen, Yoshikazu
Imai, and David Rowley Discussion:
https://postgr.es/m/9d7c5112-cb99-6a47-d3be-cf1ee6862a1d@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/428b260f87e8861ba8e58807b69d433db491c4f4

Peter Geoghegan pushed:

- Remove dead code from nbtsplitloc.c. It doesn't make sense to consider the
possibility that there will only be one candidate split point when choosing
among split points to find the split with the lowest penalty. This is a
vestige of an earlier version of the patch that became commit fab25024. Issue
spotted while rereviewing coverage of the nbtree patch series using gcov.
https://git.postgresql.org/pg/commitdiff/59ab3be9e4d3e3cdf03bada2f6fb3d71a1e42908

- Add "split after new tuple" nbtree optimization. Add additional heuristics to
the algorithm for locating an optimal split location. New logic identifies
localized monotonically increasing values in indexes with multiple columns.
When this insertion pattern is detected, page splits split just after the new
item that provoked a page split (or apply leaf fillfactor in the style of a
rightmost page split). This optimization is a variation of the long
established leaf fillfactor optimization used during rightmost page splits.
50/50 page splits are only appropriate with a pattern of truly random
insertions, where the average space utilization ends up at 65% - 70%. Without
this patch, affected cases have leaf pages that are no more than about 50%
full on average. Future insertions can never make use of the free space left
behind. With this patch, affected cases have leaf pages that are about 90%
full on average (assuming a fillfactor of 90). Localized monotonically
increasing insertion patterns are presumed to be fairly common in real-world
applications. There is a fair amount of anecdotal evidence for this. Both
pg_depend system catalog indexes (pg_depend_depender_index and
pg_depend_reference_index) are at least 20% smaller after the regression tests
are run when the optimization is available. Furthermore, many of the indexes
created by a fair use implementation of TPC-C for Postgres are consistently
about 40% smaller when the optimization is available. Note that even
pg_upgrade'd v3 indexes make use of this optimization. Author: Peter
Geoghegan Reviewed-By: Heikki Linnakangas Discussion:
https://postgr.es/m/CAH2-WzkpKeZJrXvR_p7VSY1b-s85E3gHyTbZQzR0BkJ5LrWF_A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/f21668f328c864c6b9290f39d41774cb2422f98e

- Tweak some nbtree-related code comments.
https://git.postgresql.org/pg/commitdiff/9c7fb7e6d8d0fdcabbdc32daad2159812e538bd5

Andres Freund pushed:

- tableam: Use in CREATE TABLE AS and CREATE MATERIALIZED VIEW. Previously those
directly performed a heap_insert(). Use table_insert() instead. The input
slot of those routines is not of the target relation - we could fix that by
copying if necessary, but that'd not be beneficial for performance. As those
codepaths don't access any AM specific tuple fields (say xmin/xmax), there's
no need to use an AM specific slot. Author: Andres Freund Reviewed-By:
Haribabu Kommi Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/919e48b943014b1d4ab5d83e48dbc9f8e87e7be4

- tableam: Add and use table_fetch_row_version(). This is essentially the
tableam version of heapam_fetch(), i.e. fetching a tuple identified by a tid,
performing visibility checks. Note that this different from
table_index_fetch_tuple(), which is for index lookups. It therefore has to
handle a tid pointing to an earlier version of a tuple if the AM uses an
optimization like heap's HOT. Add comments to that end. This commit removes
the stats_relation argument from heap_fetch, as it's been unused for a long
time. Author: Andres Freund Reviewed-By: Haribabu Kommi Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/9a8ee1dc650be623c32b1df103254847be974d01

- tableam: Add helper for indexes to check if a corresponding table tuples
exist. This is, likely exclusively, useful to verify that conflicts detected
in a unique index are with live tuples, rather than dead ones. Author: Andres
Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/71bdc99d0d7c3b2d73fa04fb2ff80870ce1305f7

- tableam: Add table_get_latest_tid, to wrap heap_get_latest_tid. This primarily
is to allow WHERE CURRENT OF to continue to work as it currently does. It's
not clear to me that these semantics make sense for every AM, but it works for
the in-core heap, and the out of core zheap. We can refine it further at a
later point if necessary. Author: Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/2e3da03e9ee4d6ee5cf0d1ffe0227fe6275397e1

- Remove heap_hot_search(). After 71bdc99d0d7, "tableam: Add helper for indexes
to check if a corresponding table tuples exist." there's no in-core user left.
As there's unlikely to be an external user, and such an external user could
easily be adjusted to use table_index_fetch_tuple_check(), remove
heap_hot_search(). Per complaint from Peter Geoghegan Author: Andres Freund
Discussion:
https://postgr.es/m/CAH2-Wzn0Oq4ftJrTqRAsWy2WGjv0QrJcwoZ+yqWsF_Z5vjUBFw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2ac1b2b175ea5456516e59fe2b53a1e0578a5dbd

- Compute XID horizon for page level index vacuum on primary. Previously the xid
horizon was only computed during WAL replay. That had two major problems: 1)
It relied on knowing what the table pointed to looks like. That was easy
enough before the introducing of tableam (we knew it had to be heap,
although some trickery around logging the heap relfilenodes was required).
But to properly handle table AMs we need per-database catalog access to
look up the AM handler, which recovery doesn't allow. 2) Not knowing the
xid horizon also makes it hard to support logical decoding on standbys.
When on a catalog table, we need to be able to conflict with slots that
have an xid horizon that's too old. But computing the horizon by visiting
the heap only works once consistency is reached, but we always need to be
able to detect conflicts. There's also a secondary problem, in that the
current method performs redundant work on every standby. But that's
counterbalanced by potentially computing the value when not necessary (either
because there's no standby, or because there's no connected backends). Solve
1) and 2) by moving computation of the xid horizon to the primary and by
involving tableam in the computation of the horizon. To address the
potentially increased overhead, increase the efficiency of the xid horizon
computation for heap by sorting the tids, and eliminating redundant buffer
accesses. When prefetching is available, additionally perform prefetching of
buffers. As this is more of a maintenance task, rather than something
routinely done in every read only query, we add an arbitrary 10 to the
effective concurrency - thereby using IO concurrency, when not globally
enabled. That's possibly not the perfect formula, but seems good enough for
now. Bumps WAL format, as latestRemovedXid is now part of the records, and
the heap's relfilenode isn't anymore. Author: Andres Freund, Amit Khandekar,
Robert Haas Reviewed-By: Robert Haas Discussion:
https://postgr.es/m/20181212204154.nsxf3gzqv3gesl32@alap3.anarazel.de
https://postgr.es/m/20181214014235.dal5ogljs3bmlq44@alap3.anarazel.de
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/558a9165e081d1936573e5a7d576f5febd7fb55a

- tableam: Support for an index build's initial table scan(s). To support
building indexes over tables of different AMs, the scans to do so need to be
routed through the table AM. While moving a fair amount of code, nearly all
the changes are just moving code to below a callback. Currently the range
based interface wouldn't make much sense for non block based table AMs. But
that seems aceptable for now. Author: Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/2a96909a4a8c38705163b83a81b228d5aec197f9

- Fix a few comment copy & pastos.
https://git.postgresql.org/pg/commitdiff/46bcd2af18b0406207c9dd31ff5e0dab64a76f0a

- tableam: relation creation, VACUUM FULL/CLUSTER, SET TABLESPACE. This moves
the responsibility for: - creating the storage necessary for a relation,
including creating a new relfilenode for a relation with existing storage -
non-transactional truncation of a relation - VACUUM FULL / CLUSTER's rewrite
of a table below tableam. This is fairly straight forward, with a bit of
complexity smattered in to move the computation of xid / multixid horizons
below the AM, as they don't make sense for every table AM. Author: Andres
Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/d25f519107bff602e1ebc81853fe592d020c118d

- tableam: Comment fixes. Author: Haribabu Kommi Discussion:
CAJrrPGeeYOqP3hkZyohDx_8dot4zvPuPMDBmhJ=iC85cTBNeYw(at)mail(dot)gmail(dot)com
https://git.postgresql.org/pg/commitdiff/ffa8444ce4828108e49d961cfa64e31078d978f0

- Show table access methods as such in psql's \dA. Previously we didn't display
a type for table access methods. Author: Haribabu Kommi Discussion:
CAJrrPGeeYOqP3hkZyohDx_8dot4zvPuPMDBmhJ=iC85cTBNeYw(at)mail(dot)gmail(dot)com
https://git.postgresql.org/pg/commitdiff/d3a5fc17ebdbeaec81bec1f41e304485b3292da3

- tableam: VACUUM and ANALYZE support. This is a relatively straightforward move
of the current implementation to sit below tableam. As the current analyze
sampling implementation is pretty inherently block based, the tableam analyze
interface is as well. It might make sense to generalize that at some point,
but that seems like a larger project that shouldn't be undertaken at the same
time as the introduction of tableam. Author: Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/737a292b5de296615a715ddce2b2d83d1ee245c5

- tableam: Move heap specific logic from estimate_rel_size below tableam. This
just moves the table/matview[/toast] determination of relation size to a
callback, and uses a copy of the existing logic to implement that callback for
heap. It probably would make sense to also move the index specific logic into
a callback, so the metapage handling (and probably more) can be index
specific. But that's a separate task. Author: Andres Freund Discussion:
https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/696d78469f3714a8159f7a145af1fe9179fe3291

Peter Eisentraut pushed:

- Add ORDER BY to regression test case. Apparently, the output order is
different on different endianness, per build farm member snapper.
https://git.postgresql.org/pg/commitdiff/c77e12208cd8540a209cc698373a3cfba8802639

- Initialize structure at declaration. Avoids extra memset call and cast.
Discussion:
https://www.postgresql.org/message-id/flat/7a5cbea7-b8df-e910-0f10-04014bcad701%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/572e3e6634e55accf95e2bcfb1340019c86a21dc

- Add macro to cast away volatile without allowing changes to underlying type.
This adds unvolatize(), which works just like unconstify() but for volatile.
Discussion:
https://www.postgresql.org/message-id/flat/7a5cbea7-b8df-e910-0f10-04014bcad701%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/481018f2804065cb5ed1cde75b7c0596aeef959a

- Align timestamps in pg_regress output. This way the timestamps line up in a
mix of "ok" and "FAILED" output. Author: Christoph Berg
<christoph(dot)berg(at)credativ(dot)de> Discussion:
https://www.postgresql.org/message-id/20190321115059.GF2687%40msg.df7cb.de
https://git.postgresql.org/pg/commitdiff/148cf5f462e53f374a2085b2fa8dcde944539b03

- Fix misplaced const. These instances were apparently trying to carry the const
qualifier from the arguments through the complex casts, but for that the const
qualifier was misplaced.
https://git.postgresql.org/pg/commitdiff/c8c885b7a5c8c1175288de1d8aaec3b4ae9050e1

- Use Pandoc also for plain-text documentation output. The makefile rule for the
(rarely used) plain-text output postgres.txt was still written to use lynx,
but in 96b8b8b6f9d8de4af01a77797273ad88c7a8e32e, where the INSTALL file was
switched to pandoc, the rest of the makefile support for lynx was removed, so
this was broken. Rewrite the rule to also use pandoc for postgres.txt.
https://git.postgresql.org/pg/commitdiff/2488ea7a9056ab4c0f1bbf56548e829fa40f9c4c

- doc: Move htmlhelp output to subdirectory. This makes it behave more like the
html output. That will make some subsequent changes across all output formats
easier.
https://git.postgresql.org/pg/commitdiff/477422c9d176784e90e630de3c92b4fae8590bd2

- doc: Add some images. Add infrastructure for having images in the
documentation, in SVG format. Add two images to start with. See the included
README file for instructions. Author: Jürgen Purtz <juergen(at)purtz(dot)de> Author:
Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/aaa54502-05c0-4ea5-9af8-770411a6bf4b(at)purtz(dot)de
https://git.postgresql.org/pg/commitdiff/ea55aec0a97d6cade0186df1913da2c8cb5c6f2c

- Fix vpath build. Skip doc/src/sgml/images/Makefile since the directory is not
created.
https://git.postgresql.org/pg/commitdiff/12bb35fc9b000d462b9bd6b8856e1884ef1bb3d7

- doc: Fix typo.
https://git.postgresql.org/pg/commitdiff/f3afbbdae9b6ca04a6d2b224a8dfd5959d8a17ea

- REINDEX CONCURRENTLY. This adds the CONCURRENTLY option to the REINDEX
command. A REINDEX CONCURRENTLY on a specific index creates a new index (like
CREATE INDEX CONCURRENTLY), then renames the old index away and the new index
in place and adjusts the dependencies, and then drops the old index (like DROP
INDEX CONCURRENTLY). The REINDEX command also has the capability to run its
other variants (TABLE, DATABASE) with the CONCURRENTLY option (but not
SYSTEM). The reindexdb command gets the --concurrently option. Author:
Michael Paquier, Andreas Karlsson, Peter Eisentraut Reviewed-by: Andres
Freund, Fujii Masao, Jim Nasby, Sergei Kornilov Discussion:
https://www.postgresql.org/message-id/flat/60052986-956b-4478-45ed-8bd119e9b9cf%402ndquadrant.com#74948a1044c56c5e817a5050f554ddee
https://git.postgresql.org/pg/commitdiff/5dc92b844e680c54a7ecd68de0ba53c949c3d605

- Fix incorrect code in new REINDEX CONCURRENTLY code. The previous code was
adding pointers to transient variables to a list, but by the time the list was
read, the variable might be gone, depending on the compiler. Fix it by making
copies in the proper memory context.
https://git.postgresql.org/pg/commitdiff/bb76134b08aa46cdeb0ff1e6d7b392d315529a8c

- doc: Refine README.links further. suggested by Chapman Flack
<chap(at)anastigmatix(dot)net>
https://git.postgresql.org/pg/commitdiff/c0a2ff474a47216031d1835f5878fa4f445f44fe

- doc: Fix typo. Author: Bossart, Nathan <bossartn(at)amazon(dot)com>
https://git.postgresql.org/pg/commitdiff/0267629e1889b3e1e2e6b2f1b41e8993eff86a9b

- doc: Small documentation review for REINDEX CONCURRENTLY. Author: Justin
Pryzby <pryzbyj(at)telsasoft(dot)com>
https://git.postgresql.org/pg/commitdiff/173268f4d011fcae1f909c811e264baf6c838e7a

- Small code simplification for REINDEX CONCURRENTLY. This was left over from an
earlier code structure.
https://git.postgresql.org/pg/commitdiff/6b8b5364ddd0e4d882562615c6b6c28638ade9f2

- Generated columns. This is an SQL-standard feature that allows creating
columns that are computed from expressions rather than assigned, similar to a
view or materialized view but on a column basis. This implements one kind of
generated column: stored (computed on write). Another kind, virtual (computed
on read), is planned for the future, and some room is left for it.
Reviewed-by: Michael Paquier <michael(at)paquier(dot)xyz> Reviewed-by: Pavel Stehule
<pavel(dot)stehule(at)gmail(dot)com> Discussion:
https://www.postgresql.org/message-id/flat/b151f851-4019-bdb1-699e-ebab07d2f40a(at)2ndquadrant(dot)com
https://git.postgresql.org/pg/commitdiff/fc22b6623b6b3bab3cb057ccd282c2bfad1a0b30

- doc: Fix typo. Author: Justin Pryzby <pryzby(at)telsasoft(dot)com>
https://git.postgresql.org/pg/commitdiff/ef6576f5379edfa29bb4f99880b0f76dd315dd14

Heikki Linnakangas pushed:

- Clean up the Simple-8b encoder code. Coverity complained that
simple8b_encode() might read beyond the end of the 'diffs' array, in the loop
to encode the integers. That was a false positive, because we never get into
the loop in modes 0 or 1, and the array is large enough for all the other
modes. But I admit it's very subtle, so it's not surprising that Coverity
didn't see it, and it's not very obvious to humans either. Refactor it, so
that the second loop re-computes the differences, instead of carrying them
over from the first loop in the 'diffs' array. This way, the 'diffs' array is
not needed anymore. It makes no measurable difference in performance, and
seems more straightforward this way. Also, improve the comments in
simple8b_encode(): fix the comment about its return value that was flat-out
wrong, and explain the condition when it returns EMPTY_CODEWORD better. In
the passing, move the 'selector' from the codeword's low bits to the high
bits. It doesn't matter much, but looking at the original paper, and googling
around for other Simple-8b implementations, that's how it's usually done. Per
Coverity, and Tom Lane's report off-list.
https://git.postgresql.org/pg/commitdiff/d303122eab616ccbcfb0bab0fc674bf625d17a7b

- Refactor code to print pgbench progress reports. threadRun() function is very
long and deeply-nested. Extract the code to print progress reports to a
separate function, to make it slightly easier to read. Author: Fabien Coelho
Discussion:
https://www.postgresql.org/message-id/alpine.DEB.2.21.1903101225270.17271%40lancre
https://git.postgresql.org/pg/commitdiff/9f75e3772350fb66f20a3d7f33bc94f30300d7eb

Alexander Korotkov pushed:

- Cosmetic changes for jsonpath_gram.y and jsonpath_scan.l. This commit include
formatting improvements, renamings and comments. Also, it makes
jsonpath_scan.l be more uniform with other our lexers. Firstly, states names
are renamed to more short alternatives. Secondly, <INITIAL> prefix removed
from the rules. Corresponding rules are moved to the tail, so they would
anyway work only in initial state. Author: Alexander Korotkov Reviewed-by:
John Naylor
https://git.postgresql.org/pg/commitdiff/8b17298f0b6bb2a64b55fab0339c8fd6ec2d74fb

- Get rid of backtracking in jsonpath_scan.l. Non-backtracking flex parsers work
faster than backtracking ones. So, this commit gets rid of backtracking in
jsonpath_scan.l. That required explicit handling of some cases as well as
manual backtracking for some cases. More regression tests for numerics are
added. Discussion:
https://mail.google.com/mail/u/0?ik=a20b091faa&view=om&permmsgid=msg-f%3A1628425344167939063
Author: John Naylor, Nikita Gluknov, Alexander Korotkov
https://git.postgresql.org/pg/commitdiff/1d88a75c424664cc85f307a876cde85191d27272

Robert Haas pushed:

- Add progress reporting for CLUSTER and VACUUM FULL. This uses the same
progress reporting infrastructure added in commit
c16dc1aca5e01e6acaadfcf38f5fc964a381dc62 and extends it to these additional
cases. We lack the ability to track the internal progress of sorts and index
builds so the information reported is coarse-grained for some parts of the
operation, but it still seems like a significant improvement over having
nothing at all. Tatsuro Yamada, reviewed by Thomas Munro, Masahiko Sawada,
Michael Paquier, Jeff Janes, Alvaro Herrera, Rafia Sabih, and by me. A fair
amount of polishing also by me. Discussion:
http://postgr.es/m/59A77072.3090401@lab.ntt.co.jp
https://git.postgresql.org/pg/commitdiff/6f97457e0ddd8b421ca5e483439ef0318e6fc89a

- Fix use of wrong datatype with sizeof(). OID and int are the same size, but
they are not the same thing. David Rowley Discussion:
http://postgr.es/m/CAKJS1f_MhS++XngkTvWL9X1v8M5t-0N0B-R465yHQY=TmNV0Ew@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/5857be907d5178673ce077ba71562f3c4297ee32

- Warn more strongly about the dangers of exclusive backup mode. Especially,
warn about the hazards of mishandling the backup_label file. Adjust a couple
of server messages to be more clear about the hazards associated with removing
backup_label files, too. David Steele and Robert Haas, reviewed by Laurenz
Albe, Martín Marqués, Peter Eisentraut, and Magnus Hagander. Discussion:
http://postgr.es/m/7d85c387-000e-16f0-e00b-50bf83c22127@pgmasters.net
https://git.postgresql.org/pg/commitdiff/c900c15269f0f900d666bd1b0c6df3eff5098678

- Allow existing VACUUM options to take a Boolean argument. This makes VACUUM
work more like EXPLAIN already does without changing the meaning of any
commands that already work. It is intended to facilitate the addition of
future VACUUM options that may take non-Boolean parameters or that default to
false. Masahiko Sawada, reviewed by me. Discussion:
http://postgr.es/m/CA+TgmobpYrXr5sUaEe_T0boabV0DSm=utSOZzwCUNqfLEEm8Mw@mail.gmail.com
Discussion:
http://postgr.es/m/CAD21AoBaFcKBAeL5_++j+Vzir2vBBcF4juW7qH8b3HsQY=Q6+w@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/41b54ba78e8c4d64679ba4daf82e4e2efefe1922

Álvaro Herrera pushed:

- pgbench: Remove \cset. Partial revert of commit 6260cc550b0e, "pgbench: add
\cset and \gset commands". While \gset is widely considered a useful and
necessary tool for user- defined benchmarks, \cset does not have as much
value, and its implementation was considered "not to be up to project
standards" (though I, Álvaro, can't quite understand exactly how). Therefore,
remove \cset. Author: Fabien Coelho Discussion:
https://postgr.es/m/alpine.DEB.2.21.1903230716030.18811@lancre Discussion:
https://postgr.es/m/201901101900.mv7zduch6sad@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/25ee70511ec2ccbef0ad3fe64875a4d552cdcd50

- Improve psql's \d display of foreign key constraints. When used on a partition
containing foreign keys coming from one of its ancestors, \d would (rather
unhelpfully) print the details about the pg_constraint row in the partition.
This becomes a bit frustrating when the user tries things like dropping the FK
in the partition; instead, show the details for the foreign key on the table
where it is defined. Also, when a table is referenced by a foreign key on a
partitioned table, we would show multiple "Referenced by" lines, one for each
partition, which gets unwieldy pretty fast. Modify that so that it shows only
one line for the ancestor partitioned table where the FK is defined.
Discussion: https://postgr.es/m/20181204143834.ym6euxxxi5aeqdpn@alvherre.pgsql
Reviewed-by: Tom Lane, Amit Langote, Peter Eisentraut
https://git.postgresql.org/pg/commitdiff/1af25ca0c2d9d110bd7bd5be2a14fb20cd93ee78

- psql: Schema-qualify typecast in one \d query. Bug introduced in my commit
bc87f22ef6ef
https://git.postgresql.org/pg/commitdiff/1d21ba8a9b8cb784f927a2c9c5818f8ff6779c0b

- Fix partitioned index creation bug with dropped columns. ALTER INDEX .. ATTACH
PARTITION fails if the partitioned table where the index is defined contains
more dropped columns than its partition, with this message: ERROR:
incorrect attribute map The cause was that one caller of CompareIndexInfo was
passing the number of attributes of the partition rather than the parent,
which confused the length check. Repair. This can cause pg_upgrade to fail
when used on such a database. Leave some more objects around after regression
tests, so that the case is detected by pg_upgrade test suite. Remove some
spurious empty lines noticed while looking for other cases of the same
problem. Discussion: https://postgr.es/m/20190326213924.GA2322@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/126d631222328d3def7910934bfa9cbdc99d79cc

- pgbench: doExecuteCommand -> executeMetaCommand. The new function is only in
charge of meta commands, not SQL commands. This change makes the code a little
clearer: now all the state changes are effected by advanceConnectionState. It
also removes one indent level, which makes the diff look bulkier than it
really is. Author: Fabien Coelho Reviewed-by: Kirk Jamison Discussion:
https://postgr.es/m/alpine.DEB.2.21.1811240904500.12627@lancre
https://git.postgresql.org/pg/commitdiff/9938d116331045171f37eef359992ec64c213384

Thomas Munro pushed:

- Add MacPorts support to src/test/ldap tests. Previously the test knew how to
find an OpenLDAP installation at the paths used by Homebrew. Add the MacPorts
paths too. Author: Thomas Munro Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/CA%2BhUKGKrjGS7sO4jc53gp3qipCtEvThtdP_%3DzoixgX5ZBq4Nbw%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/aa1419e63f85b23503076bb4b6413aca30b535fe

- Fix off-by-one error in txid_status(). The transaction ID returned by
GetNextXidAndEpoch() is in the future, so we can't attempt to access its
status or we might try to read a CLOG page that doesn't exist. The > vs >=
confusion probably stemmed from the choice of a variable name containing the
word "last" instead of "next", so fix that too. Back-patch to 10 where the
function arrived. Author: Thomas Munro Discussion:
https://postgr.es/m/CA%2BhUKG%2Buua_BV5cyfsioKVN2d61Lukg28ECsWTXKvh%3DBtN2DPA%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d2fd7f74ee61f41a3bd5daf2942b72cebd88f346

- Add basic infrastructure for 64 bit transaction IDs. Instead of inferring
epoch progress from xids and checkpoints, introduce a 64 bit FullTransactionId
type and use it to track xid generation. This fixes an unlikely bug where the
epoch is reported incorrectly if the range of active xids wraps around more
than once between checkpoints. The only user-visible effect of this commit is
to correct the epoch used by txid_current() and txid_status(), also visible
with pg_controldata, in those rare circumstances. It also creates some basic
infrastructure so that later patches can use 64 bit transaction IDs in more
places. The new type is a struct that we pass by value, as a form of strong
typedef. This prevents the sort of accidental confusion between TransactionId
and FullTransactionId that would be possible if we were to use a plain old
uint64. Author: Thomas Munro Reported-by: Amit Kapila Reviewed-by: Andres
Freund, Tom Lane, Heikki Linnakangas Discussion:
https://postgr.es/m/CAA4eK1%2BMv%2Bmb0HFfWM9Srtc6MVe160WFurXV68iAFMcagRZ0dQ%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2fc7af5e966043a412e8e69c135fae55a2db6d4f

- Use FullTransactionId for the transaction stack. Provide
GetTopFullTransactionId() and GetCurrentFullTransactionId(). The intended
users of these interfaces are access methods that use xids for visibility
checks but don't want to have to go back and "freeze" existing references some
time later before the 32 bit xid counter wraps around. Use a new struct to
serialize the transaction state for parallel query, because FullTransactionId
doesn't fit into the previous serialization scheme very well. Author: Thomas
Munro Reviewed-by: Heikki Linnakangas Discussion:
https://postgr.es/m/CAA4eK1%2BMv%2Bmb0HFfWM9Srtc6MVe160WFurXV68iAFMcagRZ0dQ%40mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ad308058cc8666c50b43179e64d6bb7aeb3ba169

- Fix typo. Author: Masahiko Sawada
https://git.postgresql.org/pg/commitdiff/7e69323bf72a924fd1b04a7a91da343a0cda91cf

Michaël Paquier pushed:

- Fix crash when using partition bound expressions. Since 7c079d7, partition
bounds are able to use generalized expression syntax when processed, treating
"minvalue" and "maxvalue" as specific cases as they get passed down for
transformation as a column references. The checks for infinite bounds in
range expressions have been lax though, causing crashes when trying to use
column reference names with more than one field. Here is an example causing a
crash: CREATE TABLE list_parted (a int) PARTITION BY LIST (a); CREATE TABLE
part_list_crash PARTITION OF list_parted FOR VALUES IN (somename.somename);
Note that the creation of the second relation should fail as partition bounds
cannot have column references in their expressions, so when finding an
expression which does not match the expected infinite bounds, then this commit
lets the generic transformation machinery check after it. The error message
generated in this case references as well a missing RTE, which is confusing.
This problem will be treated separately as it impacts as well default
expressions for some time, and for now only the cases where a crash can happen
are fixed. While on it, extend the set of regression tests in place for list
partition bounds and add an extra set for range partition bounds.
Reported-by: Alexander Lakhin Author: Michael Paquier Reviewed-by: Amit
Langote Discussion: https://postgr.es/m/15668-0377b1981aa1a393@postgresql.org
https://git.postgresql.org/pg/commitdiff/cdde886d36b5a4d7ad9e1d02596f7fa1c8c129e3

- Switch function current_schema[s]() to be parallel-unsafe. When invoked for
the first time in a session, current_schema() and current_schemas() can finish
by creating a temporary schema. Currently those functions are parallel-safe,
however if for a reason or another they get launched across multiple parallel
workers, they would fail when attempting to create a temporary schema as
temporary contexts are not supported in this case. The original issue has
been spotted by buildfarm members crake and lapwing, after commit c5660e0 has
introduced the first regression tests based on current_schema() in the tree.
After that, 396676b has introduced a workaround to avoid parallel plans but
that was not completely right either. Catversion is bumped. Author: Michael
Paquier Reviewed-by: Daniel Gustafsson Discussion:
https://postgr.es/m/20190118024618.GF1883@paquier.xyz
https://git.postgresql.org/pg/commitdiff/5bde1651bbba9984dc32bbeb68b329cfd3f92767

- Switch some palloc/memset calls to palloc0. Some code paths have been doing
some allocations followed by an immediate memset() to initialize the allocated
area with zeros, this is a bit overkill as there are already interfaces to do
both things in one call. Author: Daniel Gustafsson Reviewed-by: Michael
Paquier Discussion:
https://postgr.es/m/vN0OodBPkKs7g2Z1uyk3CUEmhdtspHgYCImhlmSxv1Xn6nY1ZnaaGHL8EWUIQ-NEv36tyc4G5-uA3UXUF2l4sFXtK_EQgLN1hcgunlFVKhA=@yesql.se
https://git.postgresql.org/pg/commitdiff/1983af8e899389187026cb34c1ca9d89ea986120

- Improve error handling of column references in expression transformation.
Column references are not allowed in default expressions and partition bound
expressions, and are restricted as such once the transformation of their
expressions is done. However, trying to use more complex column references
can lead to confusing error messages. For example, trying to use a two-field
column reference name for default expressions and partition bounds leads to
"missing FROM-clause entry for table", which makes no sense in their
respective context. In order to make the errors generated more useful, this
commit adds more verbose messages when transforming column references
depending on the context. This has a little consequence though: for example
an expression using an aggregate with a column reference as argument would
cause an error to be generated for the column reference, while the aggregate
was the problem reported before this commit because column references get
transformed first. The confusion exists for default expressions for a long
time, and the problem is new as of v12 for partition bounds. Still per the
lack of complaints on the matter no backpatch is done. The patch has been
written by Amit Langote and me, and Tom Lane has provided the improvement of
the documentation for default expressions on the CREATE TABLE page. Author:
Amit Langote, Michael Paquier Reviewed-by: Tom Lane Discussion:
https://postgr.es/m/20190326020853.GM2558@paquier.xyz
https://git.postgresql.org/pg/commitdiff/ecfed4a12247cf4659eee6b6ea27405e35fe57f8

- Reorganize Notes section in documentation of pg_checksums. This commit
reorders the paragraphs of the Notes section in order of importance, and
clarifies better the safe uses of pg_checksums for replication setups.
Author: Fabien Coelho Discussion:
https://postgr.es/m/alpine.DEB.2.21.1903231404280.18811@lancre
https://git.postgresql.org/pg/commitdiff/a7cc52370b2b8a0f9883e299cb07f56d702b58c2

- Skip redundant anti-wraparound vacuums. An anti-wraparound vacuum has to be by
definition aggressive as it needs to work on all the pages of a relation.
However it can happen that due to some concurrent activity an anti-wraparound
vacuum is marked as non-aggressive, which makes it redundant with a previous
run, and it is actually useless as an anti-wraparound vacuum should process
all the pages of a relation. This commit makes such vacuums to be skipped.
An anti-wraparound vacuum not aggressive can be found easily by mixing low
values of autovacuum_freeze_max_age (to control anti-wraparound) and
autovacuum_freeze_table_age (to control the aggressiveness). 28a8fa9 has
added some extra logging printing all the possible combinations of
anti-wraparound and aggressive vacuums, which now gets simplified as an
anti-wraparound vacuum also non-aggressive gets skipped. Per discussion
mainly between Andrew Dunstan, Robert Haas, Álvaro Herrera, Kyotaro Horiguchi,
Masahiko Sawada, and myself. Author: Kyotaro Horiguchi, Michael Paquier
Reviewed-by: Andrew Dunstan, Álvaro Herrera Discussion:
https://postgr.es/m/20180914153554.562muwr3uwujno75@alvherre.pgsql
https://git.postgresql.org/pg/commitdiff/2aa6e331ead7f3ad080561495ad4bd3bc7cd8913

Magnus Hagander pushed:

- Fix typo. Author: Daniel Gustafsson <daniel(at)yesql(dot)se>
https://git.postgresql.org/pg/commitdiff/05295e36ca9b40c53b7c36f5c7f75f822cd49baf

Tomáš Vondra pushed:

- Track unowned relations in doubly-linked list. Relations dropped in a single
transaction are tracked in a list of unowned relations. With large number of
dropped relations this resulted in poor performance at the end of a
transaction, when the relations are removed from the singly linked list one by
one. Commit b4166911 attempted to address this issue (particularly when it
happens during recovery) by removing the relations in a reverse order,
resulting in O(1) lookups in the list of unowned relations. This did not work
reliably, though, and it was possible to trigger the O(N^2) behavior in
various ways. Instead of trying to remove the relations in a specific order
with respect to the linked list, which seems rather fragile, switch to a
regular doubly linked. That allows us to remove relations cheaply no matter
where in the list they are. As b4166911 was a bugfix, backpatched to all
supported versions, do the same thing here. Reviewed-by: Alvaro Herrera
Discussion:
https://www.postgresql.org/message-id/flat/80c27103-99e4-1d0c-642c-d9f3b94aaa0a%402ndquadrant.com
Backpatch-through: 9.4
https://git.postgresql.org/pg/commitdiff/6ca015f9f073eef40f933f587e0b166b87261406

- Add support for multivariate MCV lists. Introduce a third extended statistic
type, supported by the CREATE STATISTICS command - MCV lists, a generalization
of the statistic already built and used for individual columns. Compared to
the already supported types (n-distinct coefficients and functional
dependencies), MCV lists are more complex, include column values and allow
estimation of much wider range of common clauses (equality and inequality
conditions, IS NULL, IS NOT NULL etc.). Similarly to the other types, a new
pseudo-type (pg_mcv_list) is used. Author: Tomas Vondra Reviewed-by: Dean
Rasheed, David Rowley, Mark Dilger, Alvaro Herrera Discussion:
https://postgr.es/m/dfdac334-9cf2-2597-fb27-f0fb3753f435@2ndquadrant.com
https://git.postgresql.org/pg/commitdiff/7300a699502fe5432b05fbc75baca534b080bebb

- Minor improvements for the multivariate MCV lists. The MCV build should always
call get_mincount_for_mcv_list(), as the there is no other logic to decide
whether the MCV list represents all the data. So just remove the (ngroups >
nitems) condition. Also, when building MCV lists, the number of items was
limited by the statistics target (i.e. up to 10000). But when deserializing
the MCV list, a different value (8192) was used to check the input, causing an
error. Simply ensure that the same value is used in both places. This should
have been included in 7300a69950, but I forgot to include it in that commit.
https://git.postgresql.org/pg/commitdiff/a63b29a1dea0ce4e95f682f9d0b36994f2fcf43e

- Fix deserialization of pg_mcv_list values. There were multiple issues in
deserialization of pg_mcv_list values. Firstly, the data is loaded from
syscache, but the deserialization was performed after ReleaseSysCache(), at
which point the data might have already disappeared. Fixed by moving the
calls in statext_mcv_load, and using the same NULL-handling code as existing
stats. Secondly, the deserialized representation used pointers into the
serialized representation. But that is also unsafe, because the data may
disappear at any time. Fixed by reworking and simplifying the deserialization
code to always copy all the data. And thirdly, when deserializing values for
types passed by value, the code simply did memcpy(d,s,typlen) which however
does not work on bigendian machines. Fixed by using
fetch_att/store_att_byval.
https://git.postgresql.org/pg/commitdiff/62bf0fb35cb370e9650981ff0333d198e955bd54

- Fix memory alignment in pg_mcv_list serialization. Blind attempt at fixing
ia64, hppa an sparc builds. The serialized representation of MCV lists did
not enforce proper memory alignment for internal fields, resulting in
deserialization issues on platforms that are more sensitive to this (ia64,
sparc and hppa). This forces a catalog version bump, because the layout of
serialized pg_mcv_list changes. Broken since 7300a699.
https://git.postgresql.org/pg/commitdiff/d85e0f366a347633f255b8d1031ab34733c5e147

- Additional fixes of memory alignment in pg_mcv_list code. Commit d85e0f366a
tried to fix memory alignment issues in serialization and deserialization of
pg_mcv_list values, but it was a few bricks shy. The arrays of uint16 indexes
in serialized items was not aligned, and the both the values and isnull flags
were using the same pointer. Per investigation by Tom Lane on gaur.
https://git.postgresql.org/pg/commitdiff/ea4e1c0e8f583f99c8caab5bcdfbbbaaed0685c0

- Fix compiler warnings in multivariate MCV code. Compiler warnings were
observed on gcc 3.4.6 (on gaur). The assert is unnecessary, as the indexes
are uint16 and so always >= 0. Reported-by: Tom Lane
https://git.postgresql.org/pg/commitdiff/ad3107b97324e0c8cf65932294115b6af9db8ded

- Fix typo. Author: John Naylor
https://git.postgresql.org/pg/commitdiff/0f5493fdf19306314e7a729750a83b26a3264412

Andrew Dunstan pushed:

- Have pg_upgrade's Makefile honor NO_TEMP_INSTALL. Backpatch to 9.5, when
pg_upgrade's location changed. Discussion:
https://postgr.es/m/5506b8fa-7dad-8483-053c-7ca7ef04f01a@2ndQuadrant.com
https://git.postgresql.org/pg/commitdiff/47b3c26642e6850e8dfa7afe01db78320b11549e

== Pending Patches ==

Fabien COELHO sent in a patch to disallow values of parameter s <= 1 in the Zipfian
distribution provided by pgbench.

Peter Eisentraut sent in a patch to fix handling of unlogged tables in FOR ALL
TABLES publications.

Jie Zhang sent in a patch to fix a bug in ALTER TABLE where a clause which would
cause a table rewrite before a clause that depends on the rewrite's having
actually occurred can cause legal statements to fail.

Panagiotis Mavrogiorgos sent in a patch to add Greek language fulltext search.

David Rowley and Alexander Kuzmenkov traded patches to remove unneeded
self-joins.

Nikita Glukhov sent in another revision of a patch to add k-nearest-neighbor
support for B-tree indexes.

Andres Freund sent in a patch to fix a bug that manifested as an Assert failure
while validating foreign keys.

Tatsuro Yamada sent in another revision of a patch to add a progress monitor for
CLUSTER.

Kyotaro HORIGUCHI sent in two more revisions of a patch to fix the WAL-skipping
feature.

Konstantin Knizhnik sent in three more revisions of a patch to add compression
options to libpq.

Hadi Moshayedi sent in another revision of a patch to fix foreign key constraint
checks for partitioned tables.

Fabien COELHO sent in two more revisions of a patch to fix the way pgbench
progress tests work.

Haribabu Kommi sent in two more revisions of a patch to support connecting to a
standby server as a priority in libpq.

Álvaro Herrera sent in three more revisions of a patch to make it possible to
monitor CREATE INDEX [CONCURRENTLY].

Nagaura Ryohei sent in twelve more revisions of a patch to add timeout
parameters for TCP.

Pavel Stěhule sent in another revision of a patch to implement schema variables.

Masahiko Sawada sent in another revision of a patch to add functions to copy
replication slots.

Thomas Munro sent in another revision of a patch to refactor the fsync mechanism
to support future SMGR implementations.

Álvaro Herrera sent in a patch to update the reference for an error that can
occur on Windows.

Pavan Deolasee sent in another revision of a patch to fix COPY ... FREEZE and
add regression tests to ensure it stays fixed.

Daniel Gustafsson sent in a patch to pg_upgrade to only allow upgrades by the
same exact version new bindir, check all used executables, and set the default
new bindir to CWD.

David Rowley sent in four more revisions of a patch to change MergeAppend to
Append under some circumstances.

Masahiko Sawada sent in two more revisions of a patch to implement block-level
parallel VACUUM.

Álvaro Herrera and Jesper Pedersen traded patches to make it possible to
reference partitioned tables with foreign keys.

Masahiko Sawada and Robert Haas traded patches to add an INDEX_CLEANUP option to
VACUUM.

Tomáš Vondra and Rafia Sabih traded patches to include non-default GUC settings
in EXPLAIN plans.

Michael Banck sent in five more revisions of a patch to make it possible to
verify page checksums online.

Surafel Temesgen and Tomáš Vondra traded patches to implement FETCH FIRST ...
WITH TIES.

Jerry Jelinek and Thomas Munro traded patches to make it possible to disallow
WAL recycling, which can be useful when the underlying filesystem is
copy-on-write.

Haribabu Kommi sent in another revision of a patch to support building with
Visual Studio 2019.

Kyotaro HORIGUCHI sent in another revision of a patch to move the stats
collector from using files to shared memory.

Dean Rasheed sent in another revision of a patch to fix bug #15708: RLS 'using'
running as wrong user when called from a view.

Aleksey Kondratov sent in another revision of a patch to pg_rewind which adds
options to use restore_command either from the command line or from instance
configuration settings like postgresql.conf.

legrand legrand and Julien Rouhaud traded patches to pass the query string to
the planner and add planning counters to pg_stat_statements.

Andrey Borodin and Heikki Linnakangas traded patches to implement GiST
verification for amcheck.

Adrien Nayrat sent in three more revisions of a patch to make it possible to log
only a sample of transactions.

Etsuro Fujita sent in two more revisions of a patch to fix some issues with plan
estimates in the PostgreSQL FDW.

Michael Banck and Michaël Paquier traded patches to implement progress reporting
for pg_verify_checksums.

Darafei Komяpa Praliaskouski sent in a patch to fix some edge cases in
autovacuum.

Andres Freund sent in another revision of a patch to make COPY use slots and
multi_insert.

Haribabu Kommi sent in another revision of a patch to avoid counting parallel
worker transactions stats.

Dmitry Dolgov sent in another revision of a patch to implement index skip scan.

Adrien Nayrat sent in a patch to fix a comment in the new sparse bitset data
structure.

Justin Pryzby sent in a patch to clean up docs for log_statement_sample_rate.

Peifeng Qiu sent in a patch to make it possible to compile PostgreSQL from
source using latest Microsoft Windows SDK.

Konstantin Knizhnik sent in a patch to make multi-tenancy more efficient.

Chapman Flack sent in two more revisions of a patch to fix the documentation
around XML.

Kyotaro HORIGUCHI sent in another revision of a patch to protect syscache from
bloating with negative cache entries.

Surafel Temesgen sent in another revision of a patch to implement FETCH FIRST
... PERCENT.

David Steele sent in another revision of a patch to add exclusive backup
deprecation notes to documentation.

Andrey Borodin sent in another revision of a patch to log bind parameter values
on error.

Haribabu Kommi sent in a patch to fix some typos in the pluggable storage docs.

Michaël Paquier sent in another revision of a patch to improve autovacuum
logging for aggressive and anti-wraparound.

Peifeng Qiu sent in a patch to speed up the build on Windows by generating
symbol definitions in a batch.

Thomas Munro sent in a patch to remove a few more lseek() calls.

Stephen Frost sent in another revision of a patch to implement GSSAPI
encryption.

Julien Rouhaud sent in a patch to add a pg_stat_checksums view.

Justin Pryzby sent in a patch to clean up the docs for 12.

Alexander Korotkov sent in another revision of a patch to add GIN support for
JSONPATH, restrict some cases in parsing numerics in JSONPATH, and throw an
error in jsonb_path_match() when result is not single boolean.

Sergey Cherkashin sent in another revision of a patch to psql to show access
methods information.

Nikolay Shaplov sent in another revision of a patch to get rid of
StdRdOptions, and use individual binary reloptions representation for each
relation kind instead.

Tom Lane sent in a patch to stop short of scanning the whole simple_rel_array
when looking only for baserels.