Skip site navigation (1) Skip section navigation (2)

Re: Large Databases

From: elein <elein(at)varlena(dot)com>
To: Joe Conway <mail(at)joeconway(dot)com>
Cc: elein <elein(at)varlena(dot)com>,pgsql-general <pgsql-general(at)postgresql(dot)org>
Subject: Re: Large Databases
Date: 2004-08-31 21:30:22
Message-ID: 20040831143022.F17217@cookie.varlena.com (view raw or flat)
Thread:
Lists: pgsql-general
I thought NFS was not recommended. Did I misunderstand this
or is there some kind of limitation to using different kinds(?)
of NFS.

Thank you for the excellent info.

--elein

On Tue, Aug 31, 2004 at 01:54:41PM -0700, Joe Conway wrote:
> elein wrote:
> >What is the linux and/or postgres limitation for very
> >large databases, if any?  We are looking at 6T-20T.
> >My understanding is that if the hardware supports it,
> >then it can be done in postgres.  But can hardware
> >support that?
> 
> I've recently been going through a project to support what will become a 
> 5 to 6 TB Postgres database (initially it will be about 300GB after 
> conversion from the source system). A few significant things I've 
> learned along the way:
> 
> 1) The linux 2.4 kernel has a block device size limit of 2 TB.
> 
> 2) The linux 2.6 kernel supports *huge* block device size -- I don't
>    have it in front of me, but IIRC it was in the peta-bytes range.
> 
> 3) xfs, jfs, and ext3 all can handle more than the 6TB we needed them to
>    handle.
> 
> 4) One of the leading SAN vendors initially claimed to be able to
>    support our desire to have a single 6TB volume. We found that when
>    pushed hard, we would get disk corruption (archives are down, but see
>    HACKERS on 8/21/04 for a message I posted on the topic). Now we are
>    being told that they don't support the linux 2.6 kernel, and
>    therefore don't support > 2TB volumes.
> 
> So the choices seem to be:
> a) Use symlinks or Postgres 8.0.0beta tablespaces to split your data
>    across multiple 2 TB volumes.
> 
> b) Use NFS mounted NAS.
> 
> We are already a big NetApp shop, so NFS mounted NAS is the direction 
> we'll likely take. It appears (from their online docs) that NetApp can 
> have individual volumes up to 16 TB. We should be confirming that with 
> them in the next day or two.
> 
> HTH,
> 
> Joe

In response to

Responses

pgsql-general by date

Next:From: Martijn van OosterhoutDate: 2004-08-31 21:34:01
Subject: Re: Multicolumn Primary Key
Previous:From: Mike NolanDate: 2004-08-31 21:24:00
Subject: Re: Single Row Table?

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group