Re: New server: SSD/RAID recommendations?

I am unsure about the performance side but, ZFS is generally very attractive to me.

Key advantages:

1) Checksumming and automatic fixing-of-broken-things on every file (not just postgres pages, but your scripts, O/S, program files).
2) Built-in lightweight compression (doesn't help with TOAST tables, in fact may slow them down, but helpful for other things). This may actually be a net negative for pg so maybe turn it off.
3) ZRAID mirroring or ZRAID5/6. If you have trouble persuading someone that it's safe to replace a RAID array with a single drive... you can use a couple of NVMe SSDs with ZFS mirror or zraid, and get the same availability you'd get from a RAID controller. Slightly better, arguably, since they claim to have fixed the raid write-hole problem.
4) filesystem snapshotting

Despite the costs of checksumming etc., I suspect ZRAID running on a fast CPU with multiple NVMe drives will outperform quite a lot of the alternatives, with great data integrity guarantees.

Haven't built one yet. Hope to, later this year. Steve, I would love to know more about how you're getting on with your NVMe disk in postgres!

Graeme.

On 07 Jul 2015, at 12:28, Mkrtchyan, Tigran <tigran(dot)mkrtchyan(at)desy(dot)de> wrote:

> Thanks for the Info.
>
> So if RAID controllers are not an option, what one should use to build
> big databases? LVM with xfs? BtrFs? Zfs?
>
> Tigran.
>
> ----- Original Message -----
>> From: "Graeme B. Bell" <graeme(dot)bell(at)nibio(dot)no>
>> To: "Steve Crawford" <scrawford(at)pinpointresearch(dot)com>
>> Cc: "Wes Vaske (wvaske)" <wvaske(at)micron(dot)com>, "pgsql-performance" <pgsql-performance(at)postgresql(dot)org>
>> Sent: Tuesday, July 7, 2015 12:22:00 PM
>> Subject: Re: [PERFORM] New server: SSD/RAID recommendations?
>
>> Completely agree with Steve.
>>
>> 1. Intel NVMe looks like the best bet if you have modern enough hardware for
>> NVMe. Otherwise e.g. S3700 mentioned elsewhere.
>>
>> 2. RAID controllers.
>>
>> We have e.g. 10-12 of these here and e.g. 25-30 SSDs, among various machines.
>> This might give people idea about where the risk lies in the path from disk to
>> CPU.
>>
>> We've had 2 RAID card failures in the last 12 months that nuked the array with
>> days of downtime, and 2 problems with batteries suddenly becoming useless or
>> suddenly reporting wildly varying temperatures/overheating. There may have been
>> other RAID problems I don't know about.
>>
>> Our IT dept were replacing Seagate HDDs last year at a rate of 2-3 per week (I
>> guess they have 100-200 disks?). We also have about 25-30 Hitachi/HGST HDDs.
>>
>> So by my estimates:
>> 30% annual problem rate with RAID controllers
>> 30-50% failure rate with Seagate HDDs (backblaze saw similar results)
>> 0% failure rate with HGST HDDs.
>> 0% failure in our SSDs. (to be fair, our one samsung SSD apparently has a bug
>> in TRIM under linux, which I'll need to investigate to see if we have been
>> affected by).
>>
>> also, RAID controllers aren't free - not just the money but also the management
>> of them (ever tried writing a complex install script that interacts work with
>> MegaCLI? It can be done but it's not much fun.). Just take a look at the
>> MegaCLI manual and ask yourself... is this even worth it (if you have a good
>> MTBF on an enterprise SSD).
>>
>> RAID was meant to be about ensuring availability of data. I have trouble
>> believing that these days....
>>
>> Graeme Bell
>>
>>
>> On 06 Jul 2015, at 18:56, Steve Crawford <scrawford(at)pinpointresearch(dot)com> wrote:
>>
>>>
>>> 2. We don't typically have redundant electronic components in our servers. Sure,
>>> we have dual power supplies and dual NICs (though generally to handle external
>>> failures) and ECC-RAM but no hot-backup CPU or redundant RAM banks and...no
>>> backup RAID card. Intel Enterprise SSD already have power-fail protection so I
>>> don't need a RAID card to give me BBU. Given the MTBF of good enterprise SSD
>>> I'm left to wonder if placing a RAID card in front merely adds a new point of
>>> failure and scheduled-downtime-inducing hands-on maintenance (I'm looking at
>>> you, RAID backup battery).
>>
>>
>>
>> --
>> Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-performance