Re: Analyzing foreign tables & memory problems

Noah Misch wrote:
>>>>> 1) Expose WIDTH_THRESHOLD in commands/vacuum.h and add
documentation
>>>>> so that the authors of foreign data wrappers are aware of the
>>>>> problem and can avoid it on their side.
>>>>> This would be quite simple.
>>
>>>> Seems reasonable. How would the FDW return an indication that a
value was
>>>> non-NULL but removed due to excess width?
>>>
>>> The FDW would return a value of length WIDTH_THRESHOLD+1 that is
>>> long enough to be recognized as too long, but not long enough to
>>> cause a problem.
>>
>> Here is a simple patch for that.
>
> It feels to me like a undue hack to ask FDW authors to synthesize such
values.
> It's easy enough for data types such as text/bytea. In general,
though,
> simple truncation may not produce a valid value of the type. That
shouldn't
> matter, since the next action taken on the value should be to discard
it, but
> it's fragile. Can we do better?
>
> Just thinking out loud, we could provide an "extern Datum
AnalyzeWideValue;"
> and direct FDW authors to use that particular datum. It could look
like a
> toasted datum of external size WIDTH_THRESHOLD+1 but bear
va_toastrelid ==
> InvalidOid. Then, if future code movement leads us to actually
examine one of
> these values, we'll get an early, hard error.

That would be very convenient indeed.

Even better would be a function
extern Datum createAnalyzeWideValue(integer width)
so that row width calculations could be more accurate.

Yours,
Laurenz Albe