Skip site navigation (1) Skip section navigation (2)

Re: silent data loss with ext4 / all current versions

From: Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
To: Andres Freund <andres(at)anarazel(dot)de>
Cc: Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com>, PostgreSQL mailing lists <pgsql-hackers(at)postgresql(dot)org>, dpage(at)pgadmin(dot)org, Guillaume Lelarge <guillaume(dot)lelarge(at)dalibo(dot)com>
Subject: Re: silent data loss with ext4 / all current versions
Date: 2016-05-12 05:58:30
Message-ID: CAB7nPqTGjtZK0OZRikFeR6tuN6AahXF8erBUtEZ2NKMxoxWb-Q@mail.gmail.com (view raw, whole thread or download thread mbox)
Thread:
Lists: pgsql-hackers
On Mon, Mar 28, 2016 at 8:25 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> I've also noticed that

Coming back to this issue because...

> a) pg_basebackup doesn't do anything about durability (it probably needs
>    a very similar patch to the one pg_rewind just received).

I think that one of the QE tests running here just got bitten by that.
A base backup was taken with pg_basebackup and more or less after a VM
was plugged off. The trick is that for pg_basebackup we cannot rely on
initdb: pg_basebackup is a client-side utility. In most of the PG
packages (Fedora, RHEL), it is put on the client-side package, where
initdb is not. So it seems to me that the correct fix is not to use
initdb -S but to have copies of fsync_parent_path, durable_rename and
fsync_fname_ext in streamutil.c, and then we reuse them for both
pg_receivexlog and pg_basebackup. At least that's less risky for
back-branches this way.

> b) nor does pg_dump[all]

I have not hacked up that yet, but I would think that we would need
one extra copy of some of those fsync_* routines in src/bin/pg_dump/.
There is another thread for that already... On master I guess we'd end
with something centralized in src/common/, but let's close the
existing holes first.

> So we're going to have another round of fsync stuff in the next set of
> releases anyway...

The sooner the better I think. Any people caring about this problem
are now limited in using initdb -S after calling pg_basebackup or
pg_dump. That's a solution, though the flushes should be contained
inside each utility.
-- 
Michael

Attachment: 0001-Issue-fsync-more-carefully-in-pg_receivexlog-and-pg_.patch
Description: application/x-download (9.7 KB)

In response to

Responses

pgsql-hackers by date

Next:From: hari.prasathDate: 2016-05-12 06:05:23
Subject: Incremental refresh of materialized view - Patch
Previous:From: Craig RingerDate: 2016-05-12 05:49:32
Subject: Re: Academic help for Postgres

Privacy Policy | About PostgreSQL
Copyright © 1996-2018 The PostgreSQL Global Development Group