Re: RFI: Extending the TOAST Pointer

Hi,

Aleksander, I'm interested in extending TOAST pointer in various ways.
64-bit TOAST value ID allows to resolve very complex issue for production
systems with large tables and heavy update rate.

I agree with Matthias that there should not be processing of TOAST pointer
internals outside TOAST macros. Currently, TOASTed value is distinguished
as VARATT_IS_EXTERNAL_ONDISK, and it should stay this way. Adding
compression requires another implementation (extension) of
VARATT_EXTERNAL because current supports only 2 compression methods -
it has only 1 bit responsible for compression method, and there is a safe
way to do so, without affecting default TOAST mechanics - we must keep
it this way for compatibility issues and not to break DB upgrade.

Also, I must remind that we should not forget about field alignment inside
the TOAST pointer.

As it was already mentioned, it seems not very reasonable trying to save
a byte or two while we are storing out-of-line values of at least 2 kb in
size.

On Mon, May 22, 2023 at 4:47 PM Aleksander Alekseev <
aleksander(at)timescale(dot)com> wrote:

> Hi,
>
> > I see your point, but I do think we should also think about why we do
> > the change.
>
> Personally at the moment I care only about implementing compression
> dictionaries on top of this, as is discussed in the corresponding
> thread [1]. I'm going to need new fields in the TOAST pointer's
> including (but not necessarily limited to) dictionary id.
>
> As I understand, Nikita is interested in implementing 64-bit TOAST
> pointers [2]. I must admit I didn't follow that thread too closely but
> I can imagine the needs are similar.
>
> Last but not least I remember somebody on the mailing list suggested
> adding ZSTD compression support for TOAST, besides LZ4. Assuming I
> didn't dream it, the proposal was rejected due to the limited amount
> of free bits in ToastCompressionId. It was argued that two possible
> combinations that are left should be treated with care and ZSTD will
> not bring enough value to the users compared to LZ4.
>
> These are 3 recent cases I could recall. This being said I think our
> solution should be generic enough to cover possible future cases
> and/or cases unknown to us yet.
>
> [1]:
> https://postgr.es/m/CAJ7c6TM7%2BsTvwREeL74Y5U91%2B5ymNobRbOmnDRfdTonq9trZyQ%40mail.gmail.com
> [2]: https://commitfest.postgresql.org/43/4296/
>
> --
> Best regards,
> Aleksander Alekseev
>

--
Regards,
Nikita Malakhov
Postgres Professional
The Russian Postgres Company
https://postgrespro.ru/