Fwd: PQgetlength vs. octet_length()

On Tue, Aug 18, 2009 at 1:48 PM, Greg Stark <gsstark(at)mit(dot)edu> wrote:

> On Tue, Aug 18, 2009 at 6:39 PM, Michael Clark<codingninja(at)gmail(dot)com>
> wrote:
> > But it seems pretty crazy that a 140meg bit of data goes to 1.3 gigs.
> Does
> > that seem a bit excessive?
>
> From what you posted earlier it looked like it was turning into about
> 500M which sounds about right. Presumably either libpq or your code is
> holding two copies of it in ram at some point in the process.
>

From what I saw, stopped at this line in my code running through gdb:
const char *valC = PQgetvalue(result, rowIndex, i);
my mem usage was 300megs. Stepping over this line it went to 1.3 gigs.
Unless there is some way to misconfigure something, I can't think how my
code could do that.
I will profile it and see if I can tell who is holding on to that memory.

> 8.5 will have an option to use a denser hex encoding but it will still
> be 2x as large as the raw data.
>

Sweet!

>
> > I avoided the binary mode because that seemed to be rather confusing when
> > having to deal with non-bytea data types. The docs make it sound like
> > binary mode should be avoided because what you get back for a datetime
> > varies per platform.
>
> There are definitely disadvantages. Generally it requires you to know
> what the binary representation of your data types is and they're not
> all well documented or guaranteed not to change in the future. I
> wouldn't recommend someone try to decode a numeric or a postgres array
> for example. And floating point numbers are platform dependent. But
> bytea is a case where it seems more natural to use binary than text
> representation.
>

To do something like this, I guess it would be best for my wrapper to being
to detect when I have a bytea column in a table and do 2 fetchs, one in text
for all other columns, and one in binary for the bytea column. Is this the
best way to handle that do you think?

Thanks,
Michael.