Re: Analyzing foreign tables & memory problems

From: "Albe Laurenz" <laurenz(dot)albe(at)wien(dot)gv(dot)at>
To: "Albe Laurenz" <laurenz(dot)albe(at)wien(dot)gv(dot)at>, "Noah Misch *EXTERN*" <noah(at)leadboat(dot)com>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Analyzing foreign tables & memory problems
Date: 2012-05-02 10:20:39
Message-ID: D960CB61B694CF459DCFB4B0128514C207D4F9DA@exadv11.host.magwien.gv.at
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

I wrote:
> Noah Misch wrote:
>>> During ANALYZE, in analyze.c, functions compute_minimal_stats
>>> and compute_scalar_stats, values whose length exceed
>>> WIDTH_THRESHOLD (= 1024) are not used for calculating statistics
>>> other than that they are counted as "too wide rows" and assumed
>>> to be all different.
>>>
>>> This works fine with regular tables;

>>> With foreign tables the situation is different. Even though
>>> values exceeding WIDTH_THRESHOLD won't get used, the complete
>>> rows will be fetched from the foreign table. This can easily
>>> exhaust maintenance_work_mem.

>>> I can think of two remedies:
>>> 1) Expose WIDTH_THRESHOLD in commands/vacuum.h and add documentation
>>> so that the authors of foreign data wrappers are aware of the
>>> problem and can avoid it on their side.
>>> This would be quite simple.

>> Seems reasonable. How would the FDW return an indication that a
value was
>> non-NULL but removed due to excess width?
>
> The FDW would return a value of length WIDTH_THRESHOLD+1 that is
> long enough to be recognized as too long, but not long enough to
> cause a problem.

Here is a simple patch for that.

Yours,
Laurenz Albe

Attachment Content-Type Size
analyze.patch application/octet-stream 2.9 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jeroen Vermeulen 2012-05-02 11:16:12 Re: extending relations more efficiently
Previous Message Hannu Krosing 2012-05-02 09:29:29 Re: JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?