Skip site navigation (1) Skip section navigation (2)

Re: [PERFORM] psql -A (unaligned format) eats too much memory

From: "Jim C(dot) Nasby" <jnasby(at)pervasive(dot)com>
To: Zoltan Boszormenyi <zboszor(at)dunaweb(dot)hu>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: [PERFORM] psql -A (unaligned format) eats too much memory
Date: 2006-06-05 15:22:59
Message-ID: 20060605152258.GT53487@pervasive.com (view raw or flat)
Thread:
Lists: pgsql-hackerspgsql-performance
Moving to -hackers

On Mon, Jun 05, 2006 at 12:32:38AM +0200, Zoltan Boszormenyi wrote:
> >I just noticed that psql's unformatted output uses too much
> >memory. Is it normal? It seems that psql draws all records
> >of a query off the server before it displays or writes the output.
> >I would expect this only with formatted output.
> >
> >Problem is, I have an export that produces 500'000+ records
> >which changes frequently. Several (20+) sites run this query
> >nightly with different parameters and download it. The SELECTs
> >that run in psql -A -t -c '...' may overlap and the query that runs
> >in less than 1.5 minutes if it's the only one at the time may take
> >3+ hours if ten such queries overlap. The time is mostly spent
> >in swapping, all psql processes take up 300+ MB, so the 1GB
> >server is brought to its knees quickly, peek swap usage is 1.8 GB.
> >I watched the progress in top and the postmaster processes finished
> >their work in about half an hour (that would still be acceptable)
> >then the psql processes started eating up memory as they read
> >the records.
> >
> >PostgreSQL 8.1.4 was used on RHEL3.
> >
> >Is there a way to convince psql to use less memory in unformatted
> >mode? I know COPY will be able to use arbitrary SELECTs
> >but until then I am still stuck with redirecting psql's output.
> 
> The answer it to use SELECT INTO TEMP and then COPY.
> Psql will use much less memory that way. But still...

I've been able to verify this on 8.1.4; psql -A -t -c 'SELECT * FROM
largetable' > /dev/null results in psql consuming vast quantities of
memory. Why is this? ISTM this is a bug...
-- 
Jim C. Nasby, Sr. Engineering Consultant      jnasby(at)pervasive(dot)com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

In response to

Responses

pgsql-performance by date

Next:From: Tom LaneDate: 2006-06-05 15:27:30
Subject: Re: [PERFORM] psql -A (unaligned format) eats too much memory
Previous:From: Jim C. NasbyDate: 2006-06-05 15:04:25
Subject: Re: Bulk loading/merging

pgsql-hackers by date

Next:From: Tom LaneDate: 2006-06-05 15:27:30
Subject: Re: [PERFORM] psql -A (unaligned format) eats too much memory
Previous:From: Teodor SigaevDate: 2006-06-05 15:12:43
Subject: Re: Connection Broken with Custom Dicts for TSearch2

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group