Skip site navigation (1) Skip section navigation (2)

Re: Analyzing foreign tables & memory problems

From: "Albe Laurenz" <laurenz(dot)albe(at)wien(dot)gv(dot)at>
To: "Albe Laurenz" <laurenz(dot)albe(at)wien(dot)gv(dot)at>,"Noah Misch *EXTERN*" <noah(at)leadboat(dot)com>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Analyzing foreign tables & memory problems
Date: 2012-05-02 10:20:39
Message-ID: D960CB61B694CF459DCFB4B0128514C207D4F9DA@exadv11.host.magwien.gv.at (view raw or flat)
Thread:
Lists: pgsql-hackers
I wrote:
> Noah Misch wrote:
>>> During ANALYZE, in analyze.c, functions compute_minimal_stats
>>> and compute_scalar_stats, values whose length exceed
>>> WIDTH_THRESHOLD (= 1024) are not used for calculating statistics
>>> other than that they are counted as "too wide rows" and assumed
>>> to be all different.
>>>
>>> This works fine with regular tables;

>>> With foreign tables the situation is different.  Even though
>>> values exceeding WIDTH_THRESHOLD won't get used, the complete
>>> rows will be fetched from the foreign table.  This can easily
>>> exhaust maintenance_work_mem.

>>> I can think of two remedies:
>>> 1) Expose WIDTH_THRESHOLD in commands/vacuum.h and add documentation
>>>    so that the authors of foreign data wrappers are aware of the
>>>    problem and can avoid it on their side.
>>>    This would be quite simple.

>> Seems reasonable.  How would the FDW return an indication that a
value was
>> non-NULL but removed due to excess width?
> 
> The FDW would return a value of length WIDTH_THRESHOLD+1 that is
> long enough to be recognized as too long, but not long enough to
> cause a problem.

Here is a simple patch for that.

Yours,
Laurenz Albe

Attachment: analyze.patch
Description: application/octet-stream (2.9 KB)

In response to

Responses

pgsql-hackers by date

Next:From: Jeroen VermeulenDate: 2012-05-02 11:16:12
Subject: Re: extending relations more efficiently
Previous:From: Hannu KrosingDate: 2012-05-02 09:29:29
Subject: Re: JSON in 9.2 - Could we have just one to_json() function instead of two separate versions ?

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group