I wrote:
> Noah Misch wrote:
>>> During ANALYZE, in analyze.c, functions compute_minimal_stats
>>> and compute_scalar_stats, values whose length exceed
>>> WIDTH_THRESHOLD (= 1024) are not used for calculating statistics
>>> other than that they are counted as "too wide rows" and assumed
>>> to be all different.
>>>
>>> This works fine with regular tables;

>>> With foreign tables the situation is different.  Even though
>>> values exceeding WIDTH_THRESHOLD won't get used, the complete
>>> rows will be fetched from the foreign table.  This can easily
>>> exhaust maintenance_work_mem.

>>> I can think of two remedies:
>>> 1) Expose WIDTH_THRESHOLD in commands/vacuum.h and add documentation
>>>    so that the authors of foreign data wrappers are aware of the
>>>    problem and can avoid it on their side.
>>>    This would be quite simple.

>> Seems reasonable.  How would the FDW return an indication that a
value was
>> non-NULL but removed due to excess width?
> 
> The FDW would return a value of length WIDTH_THRESHOLD+1 that is
> long enough to be recognized as too long, but not long enough to
> cause a problem.

Here is a simple patch for that.

Yours,
Laurenz Albe

In response to

• Re: Analyzing foreign tables & memory problems at 2012-04-30 19:01:54 from Albe Laurenz

Responses

• Re: Analyzing foreign tables & memory problems at 2012-05-13 15:45:03 from Noah Misch

pgsql-hackers by date

Next:	From: Jeroen Vermeulen	Date: 2012-05-02 11:16:12
	Subject: Re: extending relations more efficiently
Previous:	From: Hannu Krosing	Date: 2012-05-02 09:29:29
	Subject: Re: JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?