Re: Recommendations on how to combine SSD and HDD drives in bare metal PostgreSQL server

On Wed, 16 Oct 2024 17:06:24 +0300
Onni Hakala <onni(at)fyff(dot)ee> wrote:

> Hey,

Hi,

> I have a large dataset of > 100TB which would be very expensive to
> store solely into SSD drives.
>
> I have access to a server which has 2x 3.84TB NVME SSD disks and
> large array of HDD drives 8 x 22TB.
>
> Most of the data that I have in my dataset is very rarely accessed
> and is stored only for archival purposes.

I can't answer to the rest but with the configuration you describe,
you might meet some slow down because HDD speed is not what you think
it is when you reach a barrier in disks filling.

A test on a 2TB/7200RPM returns that, at the beginning of the disk it is
quite fast :
========================================================================
hdparm --offset 1g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1 GB): 504 MB in 3.00 seconds =
167.97 MB/sec

@ 50%, there's even a small gain (NB : this is a PITA, hdparm displays
GB where it uses in fact GiB:( :
=======================================================================
hdparm --offset 931g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 931 GB): 520 MB in 3.01 seconds =
172.73 MB/sec

@ 70% of the disk, the loss is only 11% :
=========================================
hdparm --offset 1304g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1304 GB): 440 MB in 3.01 seconds =
146.38 MB/sec

@ 80% of the disk the read speed has already lost 21% :
=======================================================
hdparm --offset 1490g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1490 GB): 390 MB in 3.01 seconds =
129.68 MB/sec

but @ 90% of the disk, the loss climbs @ 30.5% :
================================================
hdparm --offset 1676g --direct -t /dev/sdb1
/dev/sdb1:
geometry = 243201/255/63, sectors = 3907027087, start = 2048
Timing O_DIRECT disk reads (offset 1676 GB): 344 MB in 3.01 seconds =
114.45 MB/sec

So, for a good response time, it is best to never fill your disks
further than 75% and it is also always a good practice to avoid
some inconvenience, especially when using a COW FS, such as ZFS.

Jean-Yves

--