Skip site navigation (1) Skip section navigation (2)

Re: BIG files

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: rabt(at)dim(dot)uchile(dot)cl
Cc: pgsql-novice(at)postgresql(dot)org
Subject: Re: BIG files
Date: 2005-06-19 18:20:50
Message-ID: 18672.1119205250@sss.pgh.pa.us (view raw or flat)
Thread:
Lists: pgsql-novice
rabt(at)dim(dot)uchile(dot)cl writes:
> I work with VERY large datasets: 60 monthly tables with 700,000 rows and 99
> columns each, with mostly large numeric values (15 digits) ( NUMERIC(15,0)
> datatypes, not all filled).

> The main problem is disk space. The database files stored in postgres
> take 4 or 5 times more space than in Mysql. Just to be sure, after
> each bulk load, I performed a VACUUM FULL to reclaim any posible lost
> space, but nothing gets reclaimed. My plain text dump files with
> INSERTS are just 150 Mb in size, while the files in Postgres directory
> are more than 1 Gb each!!

Hmm ... it's conventional wisdom that a PG data file will be larger than
an equivalent text file, but you seem to have a much worse case than I'd
expect.  As Bruno noted, BIGINT sounds like a better choice than NUMERIC
for your data, but even with NUMERIC I don't see where the bloat is
coming from.  A 15-digit NUMERIC value ought to occupy 16 bytes on disk
(8 bytes of overhead + 4 base-10000 digits at 2 bytes each), so that
ought to be just about one-for-one with the text representation.  And
with 99 of those per row, the 32-bytes-per-row row header overhead isn't
what's killing you either.

Could we see the exact table declaration (including indexes) and a few
rows of sample data?

			regards, tom lane

In response to

Responses

pgsql-novice by date

Next:From: Prasad devDate: 2005-06-20 00:27:41
Subject: Raise Notice
Previous:From: Stephan SzaboDate: 2005-06-19 16:33:01
Subject: Re: BIG files

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group