Skip site navigation (1) Skip section navigation (2)

Re: literal limits in 8.3

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Sam Mason <sam(at)samason(dot)me(dot)uk>
Cc: pgsql-general(at)postgresql(dot)org, pgsql-hackers(at)postgresql(dot)org
Subject: Re: literal limits in 8.3
Date: 2008-11-24 19:46:45
Message-ID: 4969.1227556005@sss.pgh.pa.us (view raw or flat)
Thread:
Lists: pgsql-generalpgsql-hackers
Sam Mason <sam(at)samason(dot)me(dot)uk> writes:
> On Mon, Nov 24, 2008 at 10:45:42AM -0500, Tom Lane wrote:
>> Well, I can't reproduce that here.  Something strange about your
>> configuration maybe?

> Not that I know of.  I've just created a test cluster to make sure and I
> get the same behaviour.

Hmm ... the third machine I tried was able to reproduce the problem.

What it boils down to is lack of error checking in psql (not the
backend).  Specifically, we fail to enlarge the output buffer for
psqlscan.l, which causes appendBinaryPQExpBuffer to silently not insert
the chunk it's currently being passed.  Which you might think would be
some random subset of the input string, leading to a parse error on
the backend side --- but no, this is the output of a lexical scan which
means what is dropped is exactly the contents of the multi-megabyte
string literal, not less or more.  And then later insertions work fine
since *they* aren't provoking an out-of-memory problem.  So eventually
the backend receives
	INSERT INTO test (col) VALUES ('');
which of course it finds nothing wrong with.

This is sort of a PITA to fix :-(.  The easiest solution from the point
of view of psql would be to have realloc failure just print "out of
memory" and exit(1), but pqexpbuffer.c is part of libpq and so it's not
too reasonable for it to do that.  And we have also got to think about
the prospect of similarly non-random lossage in other uses of
PQexpbuffer, anyhow.

The least API-damaging solution I can think of is to add an error
indicator flag to PQexpbuffer, comparable to ferror() on stdio files.
Callers would have to check this after loading up a PQexpbuffer if
they wanted to be sure there was no memory overrun.  But that seems
pretty fragile, and it wouldn't be back-patchable.

A variant on that is to clear the buffer and insert "out of memory"
in this scenario, but that's not too pleasant either.

Better ideas anyone?

			regards, tom lane

In response to

Responses

pgsql-hackers by date

Next:From: Greg Sabino MullaneDate: 2008-11-24 20:08:00
Subject: Re: literal limits in 8.3
Previous:From: Tom LaneDate: 2008-11-24 19:03:41
Subject: Re: Visibility map, partial vacuums

pgsql-general by date

Next:From: Greg Sabino MullaneDate: 2008-11-24 20:08:00
Subject: Re: literal limits in 8.3
Previous:From: Grzegorz JaƛkiewiczDate: 2008-11-24 17:13:56
Subject: Re: filter statements in logs

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group