Re: Pluggable Storage - Andres's take

From: Andres Freund <andres(at)anarazel(dot)de>
To: Haribabu Kommi <kommi(dot)haribabu(at)gmail(dot)com>
Cc: Asim R P <apraveen(at)pivotal(dot)io>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>, aagrawal(at)pivotal(dot)io, Dmitry Dolgov <9erthalion6(at)gmail(dot)com>, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>, Alexander Korotkov <a(dot)korotkov(at)postgrespro(dot)ru>
Subject: Re: Pluggable Storage - Andres's take
Date: 2019-01-21 03:01:15
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


(resending with compressed attachements, perhaps that'll go through)

On 2018-12-10 18:13:40 -0800, Andres Freund wrote:
> On 2018-11-26 17:55:57 -0800, Andres Freund wrote:
> > FWIW, now that oids are removed, and the tuple table slot abstraction
> > got in, I'm working on rebasing the pluggable storage patchset ontop of
> > that.
> I've pushed a version to that to the git tree, including a rebased
> version of zheap:

I've pushed the newest, substantially revised, version to the same
repository. Note, that while the newest pluggable-zheap version is newer
than my last email, it's not based on the latest version, and the
pluggable-zheap development is now happening in the main zheap

> My next steps are:
> - make relation creation properly pluggable
> - remove the typedefs from tableam.h, instead move them into the
> TableAmRoutine struct.
> - Move rs_{nblocks, startblock, numblocks} out of TableScanDescData
> - Move HeapScanDesc and IndexFetchHeapData out of relscan.h
> - remove ExecSlotCompare(), it's entirely unrelated to these changes imo
> (and in the wrong place)

These are done.

> - split pluggable storage patchset, to commit earlier:
> - EvalPlanQual slotification
> - trigger slotification
> - split of IndexBuildHeapScan out of index.c

The patchset is now pretty granularly split into individual pieces.
There's two commits that might be worthwhile to split up further:

1) The commit introducing table_beginscan et al, currently also
introduces indexscans through tableam.
2) The commit introducing table_(insert|delete|update) also includes
table_lock_tuple(), which in turn changes a bunch of EPQ related
code. It's probably worthwhile to break that out.

I tried to make each individual commit make some sense, and pass all
tests on its own. That requires some changes that are then obsolted
in a later commit, but it's not as much as I feared.

> - rename HeapUpdateFailureData et al to not reference Heap

I've not done that, I decided it's best to do that after all the work
has gone in.

> - See if the slot in SysScanDescData can be avoided, it's not exactly
> free of overhead.

After reconsidering, I don't think it's worth doing so.

There's pretty substantial changes in this series, besides the things
mentioned above:

- I re-introduced parallel scan into pluggable storage, but added a set
of helper functions to avoid having to duplicate the current block
based logic from heap. That way it can be shared between most/all
block based AMs
- latestRemovedXid handling is moved into the table-AM, that's required
for correct replay on Hot-Standby, where we do not know the AM of the
- the whole truncation and relation creation code has been overhauled
- the order of functions in tableam.h, heapam_handler.c etc has been
made more sensible
- a number of callbacks have been obsoleted (relation_sync,
relation_create_init_fork, scansetlimits)
- A bunch of prerequisite work has been merged
- (heap|relation)_(open|openrv|close) have been split into their own
- To avoid having to care about the bulk-insert flags code that uses a
bulk-insert now unconditionally calls table_finish_bulk_insert(). The
AM then internally can decide what it needs to do in case of
e.g. HEAP_INSERT_SKIP_WAL. Zheap currently for example doesn't
implement that (because UNDO handling is complicated), and this way it
can just ignore the option, without needing call-site code for that.
- A *lot* of cleanups

- merge psql / pg_dump support by Dmitry
- consider removing scan_update_snapshot
- consider removing table_gimmegimmeslot()
- add substantial docs for every callback
- consider revising the current table_lock_tuple() API, I'm not quite
convinced that's right
- reconsider heap_fetch() API changes, causes unnecessary pain
- polish the split out trigger and EPQ changes, so they can be merged

I plan to merge the first few commits pretty soon (as largely announced
in related threads).

While I saw an initial attempt at writing smgl docs for the table AM
API, I'm not convinced that's the best approach. I think it might make
more sense to have high-level docs in sgml, but then do all the
per-callback docs in tableam.h.


Andres Freund

Attachment Content-Type Size
v12-0001-WIP-Introduce-access-table.h-access-relation.h.patch.gz application/x-patch-gzip 5.2 KB
v12-0002-Replace-heapam.h-includes-with-relation.h-table..patch.gz application/x-patch-gzip 7.0 KB
v12-0003-Replace-uses-of-heap_open-et-al-with-table_open-.patch.gz application/x-patch-gzip 73.9 KB
v12-0004-Remove-superfluous-tqual.h-includes.patch.gz application/x-patch-gzip 4.4 KB
v12-0005-WIP-rename-tqual.c-to-heapam_visibility.c.patch.gz application/x-patch-gzip 3.8 KB
v12-0006-WIP-change-snapshot-type-to-enum-rather-than-cal.patch.gz application/x-patch-gzip 4.3 KB
v12-0007-Move-generic-snapshot-related-code-from-tqual.h-.patch.gz application/x-patch-gzip 5.0 KB
v12-0008-Move-heap-visibility-routines-to-heapam.h.patch.gz application/x-patch-gzip 3.7 KB
v12-0009-Rephrase-references-to-time-qualification.patch.gz application/x-patch-gzip 1.5 KB
v12-0010-WIP-Extend-tuples-when-getting-them-from-a-slot.patch.gz application/x-patch-gzip 1.0 KB
v12-0011-WIP-Move-page-initialization-from-RelationAddExt.patch.gz application/x-patch-gzip 2.5 KB
v12-0012-WIP-ForceStore-HeapTupleDatum.patch.gz application/x-patch-gzip 943 bytes
v12-0013-slot-type-fixes.patch.gz application/x-patch-gzip 1.3 KB
v12-0014-Rename-RelationData.rd_amroutine-to-rd_indam.patch.gz application/x-patch-gzip 4.7 KB
v12-0015-Add-ExecStorePinnedBufferHeapTuple.patch.gz application/x-patch-gzip 1.8 KB
v12-0016-Store-HeapTupleData-in-Buffer-HeapTupleTableSlot.patch.gz application/x-patch-gzip 878 bytes
v12-0017-Buffer-tuples-may-be-virtualized.patch.gz application/x-patch-gzip 1.0 KB
v12-0018-slot-tableoid-tid-support.patch.gz application/x-patch-gzip 1.3 KB
v12-0019-Set-tableoid-in-a-bunch-of-places.patch.gz application/x-patch-gzip 1.8 KB
v12-0020-WIP-Slotified-triggers.patch.gz application/x-patch-gzip 19.2 KB
v12-0021-WIP-Slotify-EPQ.patch.gz application/x-patch-gzip 7.7 KB
v12-0022-tableam-introduce-minimal-infrastructure.patch.gz application/x-patch-gzip 14.0 KB
v12-0023-tableam-Introduce-and-use-begin-endscan-and-do-i.patch.gz application/x-patch-gzip 44.8 KB
v12-0024-tableam-Inquire-slot-type-from-AM-rather-than-ha.patch.gz application/x-patch-gzip 6.1 KB
v12-0025-tableam-introduce-slot-based-table-getnext-and-u.patch.gz application/x-patch-gzip 6.7 KB
v12-0026-tableam-Add-insert-delete-update-lock_tuple.patch.gz application/x-patch-gzip 20.2 KB
v12-0027-tableam-Add-fetch_row_version.patch.gz application/x-patch-gzip 3.6 KB
v12-0028-tableam-Add-use-tableam_fetch_follow_check.patch.gz application/x-patch-gzip 1.5 KB
v12-0029-tableam-Add-table_get_latest_tid.patch.gz application/x-patch-gzip 1.7 KB
v12-0030-tableam-multi_insert-and-slotify-COPY.patch.gz application/x-patch-gzip 6.0 KB
v12-0031-tableam-finish_bulk_insert.patch.gz application/x-patch-gzip 1.9 KB
v12-0032-tableam-slotify-CREATE-TABLE-AS-and-CREATE-MATER.patch.gz application/x-patch-gzip 1.5 KB
v12-0033-tableam-index-builds.patch.gz application/x-patch-gzip 16.0 KB
v12-0034-tableam-relation-creation-VACUUM-FULL-CLUSTER-SE.patch.gz application/x-patch-gzip 15.4 KB
v12-0035-tableam-VACUUM-and-ANALYZE.patch.gz application/x-patch-gzip 5.4 KB
v12-0036-tableam-planner-size-estimation.patch.gz application/x-patch-gzip 3.9 KB
v12-0037-tableam-Sample-Scan-Support.patch.gz application/x-patch-gzip 7.2 KB
v12-0038-tableam-bitmap-heap-scan.patch.gz application/x-patch-gzip 5.0 KB
v12-0039-tableam-remaining-stuff.patch.gz application/x-patch-gzip 3.9 KB
v12-0040-WIP-Move-xid-horizon-computation-for-page-level-.patch.gz application/x-patch-gzip 6.2 KB
v12-0041-tableam-Add-function-to-determine-newest-xid-amo.patch.gz application/x-patch-gzip 1.2 KB
v12-0042-tableam-Fetch-tuples-for-triggers-EPQ-using-a-pr.patch.gz application/x-patch-gzip 2.2 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2019-01-21 03:16:01 Re: Thread-unsafe coding in ecpg
Previous Message Tsunakawa, Takayuki 2019-01-21 02:57:34 RE: Thread-unsafe coding in ecpg