Re: Optimize external TOAST storage

On Tue, Mar 22, 2022 at 02:42:53PM -0700, Nathan Bossart wrote:
> On Tue, Mar 22, 2022 at 04:34:05PM -0400, Robert Haas wrote:
>> Then, too, I'm not very confident about the usefulness of EXTENDED,
>> EXTERNAL, and MAIN. I think it's useful to be able to categorically
>> disable compression (as EXTERNAL does), but you can't categorically
>> disable out-of-line storage because the value can be bigger than the
>> page, so MAIN vs. EXTENDED is just changing the threshold for the use
>> of out-of-line storage. However, it does so in a way that IMHO is not
>> particularly intuitive, which goes back to my earlier point about the
>> algorithm seeming wacky, and it's in any case unclear why we should
>> offer exactly two possible thresholds and not any of the intermediate
>> values.
>
> I agree with all of this. Adding configurability for each constant might
> help folks in the short term, but using these knobs probably requires quite
> a bit of expertise in Postgres internals as well as a good understanding of
> your data. I think we ought to revist TOAST configurability from a user
> perspective. IOW what can be chosen automatically, and how do we enable
> users to effectively configure the settings that cannot be chosen
> automatically? IMO this is a worthwhile conversation to have as long as it
> doesn't stray too far into the "let's rewrite TOAST" realm. I think there
> is always going to be some level of complexity with stuff like TOAST, but
> there are probably all sorts of ways to simplify/improve it also.

I agree with this feeling. TOAST has already too many configuration
parameters that have their own way of behaving slightling differently.
If we could reduce this number rather than increasing it, the better.
I would be also curious to see how much those parameters become
relevant with more compression options possible with toast values.
--
Michael