Re: Pluggable toaster

Hi,

Pluggable TOAST is provided as an interface to allow developers to plug
in custom TOAST mechanics. It does not determines would it be universal
Toaster or one data type, but syntax for universal Toaster is out of scope
for this patchset.

>True, but besides Jsonb Toaster one can implement a universal one as
>well (your own example: encryption, or a Toaster that bypasses a 1 GB
>value limitation). However we can probably keep this out of scope of
>this particular patchset for now. As mentioned before this is going to
>be just an additional syntax for the user convenience.

To transparently bypass the 1 Gb limit you have to increase size of data
VARLENA type is able to hold. This is out if scope for this patchset too,
but as I mentioned before, there are means to do this with Pluggable
TOAST using toast and detoast iterators.

>Compression is actually a part of the four-stage TOAST algorithm. So
>what you're doing is using the default TOAST most of the time, but if
>the storage strategy is EXTERNAL and a custom TOASTer is specified
>then you use a type-aware "TOASTer".

We provide several custom Toasters for particular types of data causing
a lot of problems in storage. The main idea behind Pluggable TOAST is
using special data-aware Toasters where it is needed.
Extended storage mode supports only 2 compression algorithms, though
there are more efficient ones for different types of data.

>If the goal is to implement true "Pluggable TOASTers" then the TOAST
>should be substituted entirely. If the goal is to implement type-aware
>sub-TOASTers we should be honest about this and call it otherwise:
>e.g. "Type-aware TOASTer" or perhaps "subTOASTer". Additionally in
>this case there should be no validate() method since this is going to
>work only with the default heapam that implements the default TOASTer.

>So to clarify, the goal is to deliver true "Pluggable TOASTers" or
>only "type-aware sub-TOASTers"?

Pluggable TOAST does not supposes complete substitution of existing
TOAST mechanics - this is out of scope for this patchset. It proposes
means to substitute it or plug in additional custom TOAST mechanics
for particular data types.

>I don't see any reason why the semantics for Toasters should be any
>different from user-defined types implemented in an extension. If
>there are columns that use a given Toaster we should forbid DROPping
>the extension. Otherwise "DROP extension" should succeed. No one says
>that this should be a fast operation but it should be possible.

I'm currently working on a revision of Pluggable TOAST that would make
dropping Toaster possible if there is no data TOASTed with it, along with
several other major changes. It will be available in this (I hope so) or the
following, if I won't make it in time, commitfest.

On Fri, Nov 4, 2022 at 12:35 PM Aleksander Alekseev <
aleksander(at)timescale(dot)com> wrote:

> Hi again,
>
> > [1]:
> https://www.postgresql.org/message-id/flat/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22=5xVBg7S4vr5rQ(at)mail(dot)gmail(dot)com
>
> I added a link but forgot to use it, d'oh!
>
> Please note that if the goal is to merely implement "type-aware
> sub-TOASTers" then compression dictionaries [1] arguably provide the
> same value with MUCH less complexity.
>
> --
> Best regards,
> Aleksander Alekseev
>

--
Regards,
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/