Re: Is MaxHeapAttributeNumber a reasonable restriction for foreign-tables?

Hello,

On Thu, Feb 4, 2021 at 4:24 PM Kohei KaiGai <kaigai(at)heterodb(dot)com> wrote:
> I noticed that CheckAttributeNamesTypes() prevents to create a table that has
> more than MaxHeapAttributeNumber (1600) columns, for foreign-table also.
> IIUC, this magic number comes from length of the null-bitmap can be covered
> with t_hoff in HeapTupleHeaderData.
> For heap-tables, it seems to me a reasonable restriction to prevent overrun of
> null-bitmap. On the other hand, do we have proper reason to apply same
> restrictions on foreign-tables also?
>
> Foreign-tables have their own unique internal data structures instead of
> the PostgreSQL's heap-table, and some of foreign-data can have thousands
> attributes in their structured data.
> I think that MaxHeapAttributeNumber is a senseless restriction for foreign-
> tables. How about your opinions?

My first reaction to this was a suspicion that the
MaxHeapAttributeNumber limit would be too ingrained in PostgreSQL's
architecture to consider this matter lightly, but actually browsing
the code, that may not really be the case. Other than
src/backend/access/heap/*, here are the places that check it:

catalog/heap.c: CheckAttributeNamesTypes() that you mentioned:

/* Sanity check on column count */
if (natts < 0 || natts > MaxHeapAttributeNumber)
ereport(ERROR,
(errcode(ERRCODE_TOO_MANY_COLUMNS),
errmsg("tables can have at most %d columns",
MaxHeapAttributeNumber)));

tablecmds.c: MergeAttributes():

/*
* Check for and reject tables with too many columns. We perform this
* check relatively early for two reasons: (a) we don't run the risk of
* overflowing an AttrNumber in subsequent code (b) an O(n^2) algorithm is
* okay if we're processing <= 1600 columns, but could take minutes to
* execute if the user attempts to create a table with hundreds of
* thousands of columns.
*
* Note that we also need to check that we do not exceed this figure after
* including columns from inherited relations.
*/
if (list_length(schema) > MaxHeapAttributeNumber)
ereport(ERROR,
(errcode(ERRCODE_TOO_MANY_COLUMNS),
errmsg("tables can have at most %d columns",
MaxHeapAttributeNumber)));

tablecmds.c: ATExecAddColumn():

/* Determine the new attribute's number */
newattnum = ((Form_pg_class) GETSTRUCT(reltup))->relnatts + 1;
if (newattnum > MaxHeapAttributeNumber)
ereport(ERROR,
(errcode(ERRCODE_TOO_MANY_COLUMNS),
errmsg("tables can have at most %d columns",
MaxHeapAttributeNumber)));

So, unless I am terribly wrong, we may have a shot at revisiting the
decision that would have set this limit.

--
Amit Langote
EDB: http://www.enterprisedb.com