Re: allocation limit for encoding conversion

Hi,

On 2019-08-16 17:31:49 -0400, Tom Lane wrote:
> Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
> > Somebody ran into issues when generating large XML output (upwards of
> > 256 MB) and then sending via a connection with a different
> > client_encoding. This occurs because we pessimistically allocate 4x as
> > much memory as the string needs, and we run into the 1GB palloc
> > limitation. ISTM we can do better now by using huge allocations, as per
> > the preliminary attached patch (which probably needs an updated overflow
> > check rather than have it removed altogether); but at least it is able
> > to process this query, which it wasn't without the patch:
>
> > select query_to_xml(
> > 'select a, cash_words(a::text::money) from generate_series(0, 2000000) a',
> > true, false, '');
>
> I fear that allowing pg_do_encoding_conversion to return strings longer
> than 1GB is just going to create failure cases somewhere else.
>
> However, it's certainly true that 4x growth is a pretty unlikely worst
> case. Maybe we could do something like
>
> 1. If string is short (say up to a few megabytes), continue to do it
> like now. This avoids adding overhead for typical cases.
>
> 2. Otherwise, run some lobotomized form of encoding conversion that
> just computes the space required (as an int64, I guess) without saving
> the result anywhere.
>
> 3. If space required > 1GB, fail.
>
> 4. Otherwise, allocate just the space required, and convert.

It's probably too big a hammer for this specific case, but I think at
some point we ought to stop using fixed size allocations for this kind
of work. Instead we should use something roughly like our StringInfo,
except that when exceeding the current size limit, the overflowing data
is stored in a separate allocation. And only once we actually need the
data in a consecutive form, we allocate memory that's large enough to
store the all the separate allocations in their entirety.

Greetings,

Andres Freund