Re: RFI: Extending the TOAST Pointer

From: Matthias van de Meent <boekewurm+postgres(at)gmail(dot)com>
To: Aleksander Alekseev <aleksander(at)timescale(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Nikita Malakhov <hukutoc(at)gmail(dot)com>
Subject: Re: RFI: Extending the TOAST Pointer
Date: 2023-05-18 12:05:54
Message-ID: CAEze2WibtT+EdGscamqDm3+7qauWi_S6nVUhbK_qg1rkqN_48w@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Thu, 18 May 2023 at 12:52, Aleksander Alekseev
<aleksander(at)timescale(dot)com> wrote:
>
> Hi Nikita,
>
> > this part of the PostgreSQL screams to be revised and improved
>
> I completely agree. The problem with TOAST pointers is that they are
> not extendable at the moment which prevents adding new compression
> algorithms (e.g. ZSTD), new features like compression dictionaries
> [1], etc. I suggest we add extensibility in order to solve this
> problem for the foreseeable future for everyone.
>
> > where Custom TOAST Pointer is distinguished from Regular one by va_flag field
> > which is a part of varlena header
>
> I don't think that varlena header is the best place to distinguish a
> classical TOAST pointer from an extended one. On top of that I don't
> see any free bits that would allow adding such a flag to the on-disk
> varlena representation [2].
>
> The current on-disk TOAST pointer representation is following:
>
> ```
> typedef struct varatt_external
> {
> int32 va_rawsize; /* Original data size (includes header) */
> uint32 va_extinfo; /* External saved size (without header) and
> * compression method */
> Oid va_valueid; /* Unique ID of value within TOAST table */
> Oid va_toastrelid; /* RelID of TOAST table containing it */
> } varatt_external;
> ```

No, that's inaccurate. The complete on-disk representation of a varatt is

{
uint8 va_header; /* Always 0x80 or 0x01 */
uint8 va_tag; /* Type of datum */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Type-dependent
data, for toasted values that's currently only a varatt_external */
} varattrib_1b_e;

With va_tag being filled with one of the vartag_external values:

typedef enum vartag_external
{
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
VARTAG_ONDISK = 18
} vartag_external;

This enum still has many options to go before it exceeds the maximum
of the uint8 va_tag field. Therefore, I don't think we have no disk
representations left, nor do I think we'll need to add another option
to the ToastCompressionId enum.
As an example, we can add another VARTAG option for dictionary-enabled
external toast; like what the pluggable toast patch worked on. I think
we can salvage some ideas from that patch, even if the main idea got
stuck.

Kind regards,

Matthias van de Meent
Neon Inc.

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andrew Dunstan 2023-05-18 12:20:13 Re: No buildfarm animals are running both typedefs and --with-llvm
Previous Message Aleksander Alekseev 2023-05-18 11:17:02 Re: [PATCH] Allow Postgres to pick an unused port to listen