pg_basebackup fsyncs some files despite --no-sync (was: Adding CI to our tree)

From: Andres Freund <andres(at)anarazel(dot)de>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Justin Pryzby <pryzby(at)telsasoft(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Andrew Dunstan <andrew(at)dunslane(dot)net>, pgsql-hackers(at)postgresql(dot)org, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>, Melanie Plageman <melanieplageman(at)gmail(dot)com>, Peter Eisentraut <peter(dot)eisentraut(at)enterprisedb(dot)com>, Daniel Gustafsson <daniel(at)yesql(dot)se>
Subject: pg_basebackup fsyncs some files despite --no-sync (was: Adding CI to our tree)
Date: 2022-01-21 20:00:57
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


On 2022-01-18 20:16:46 -0800, Andres Freund wrote:
> I noticed a few other sources of "unnecessary" fsyncs. The most frequent
> being the durable_rename() of backup_manifest in pg_basebackup.c. Manifests are
> surprisingly large, 135k for a freshly initdb'd cluster.

Robert, I assume the fsync for manifests isn't ignoring --no-sync for a
particular reason?

The attached patch adds no-sync handling to the manifest rename, as well as
one case in the directory wal method.

It's a bit painful that we have to have code like

if (dir_data->sync)
r = durable_rename(tmppath, tmppath2);
if (rename(tmppath, tmppath2) != 0)
pg_log_error("could not rename file \"%s\" to \"%s\": %m",
tmppath, tmppath2);
r = -1;

It seems like it'd be better to set it up so that durable_rename() could
decide internally wether to fsync, or have a wrapper around durable_rename()?

> There's an fsync in walmethods.c:tar_close() that sounds intentional, but I
> don't really understand what the comment:
> /* Always fsync on close, so the padding gets fsynced */
> if (tar_sync(f) < 0)

tar_sync() actually checks for tar_data->sync, so it doesn't do an
fsync. Arguably the comment is a bit confusing, but ...


Andres Freund

Attachment Content-Type Size
v1-0001-pg_basebackup-Skip-a-few-more-fsyncs-if-no-sync-i.patch text/x-diff 2.1 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Greg Stark 2022-01-21 20:06:24 Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations
Previous Message Tom Lane 2022-01-21 19:57:12 Re: Push down time-related SQLValue functions to foreign server