Skip site navigation (1) Skip section navigation (2)

Re: fsyncing data to disk

From: Florian Pflug <fgp(at)phlo(dot)org>
To: Nulik Nol <nuliknol(at)gmail(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: fsyncing data to disk
Date: 2011-09-09 18:46:51
Message-ID: 69CE224B-5C78-4CAE-B955-3725F66E8F92@phlo.org (view raw or flat)
Thread:
Lists: pgsql-hackers
On Sep9, 2011, at 20:15 , Nulik Nol wrote:
> this is not exactly a Postgresql question, but an input from hackers
> list like this would be invaluable for me.
> I am coding my own database engine, and I decided to do not implement
> transaction engine because it implies too much code.
> But to achieve the Durability of ACID I need a 100% reliable write to
> disk. By design no record in my DB will be larger than 512 bytes, so I
> am using the page size of 512 bytes,

Beware that there *are* disks with block sizes other than 512 bytes. For
example, at least for 2.5" disks, 4096 bytes/block is becoming quite
common these days.

> that matches the size of the disk
> block, so every write() I will execute with the following fdatasync()
> call will be 100% written, is that correct? It won't make a 300 byte
> write if I tell it to write 512 and the power goes off or will it?

Since error correction is done per-block, it's very unlikely that you'd see
only 300 of the 512 bytes overwritten - the drive would detect uncorrectable
data corruption and report an error instead. Whether that error is reported back
to the application as an IO error or as a zeroed-out block probably depends on
the OS.

What you actually seem to want is a stronger all-or-nothing guarantee which
precludes the error case. AFAIK, most disk drives kinda-of do that, because
the various capacitors which stabilize the power supply usually hold enough
charge to complete a write once it's started, and because they stop operating
if the power drops below some threshold. But I doubt that they provide any
hard guarantees in this area, I guess it's more of a best-effort thing.

To get hard guarantees, you'll need to use a RAID controller with a
battery-backed cache. Or use a journal/WAL like postgres (and most filesystems)
do, and protect journal/WAL entries with a checksum to detect partially written
entries.

> I am going to use the whole partition device for the DB (like /dev/sda1)
> , so no filesystem code will be used. Also I am using asynchronous IO
> (the aio_read and aio_write) and I don't know if they can be combined
> with the fdatasync() syscall?

Someone else (maybe the POSIX spec?) must answer that as I know very little
about asynchronous IO.

best regards,
Florian Pflug


In response to

Responses

pgsql-hackers by date

Next:From: Florian PflugDate: 2011-09-09 18:59:43
Subject: Re: postgresql.conf archive_command example
Previous:From: Kevin GrittnerDate: 2011-09-09 18:38:08
Subject: Re: Patch to improve reliability of postgresql on linux nfs

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group