Re: design for parallel backup

On Sat, May 2, 2020 at 10:36 PM Andres Freund <andres(at)anarazel(dot)de> wrote:
> I changed Robert's test program to optionall fallocate,
> sync_file_range(WRITE), posix_fadvise(DONTNEED), to avoid a large
> footprint in the page cache. The performance
> differences are quite substantial:
>
> gcc -Wall -ggdb ~/tmp/write_and_fsync.c -o /tmp/write_and_fsync && \
> rm -ff /srv/dev/bench/test* && echo 3 |sudo tee /proc/sys/vm/drop_caches && \
> /tmp/write_and_fsync --sync_file_range=0 --fallocate=0 --fadvise=0 --filesize=$((400*1024*1024*1024)) /srv/dev/bench/test1
>
> running test with: numprocs=1 filesize=429496729600 blocksize=8192 fallocate=0 sfr=0 fadvise=0
> [/srv/dev/bench/test1][11450] open: 0, fallocate: 0 write: 214, fsync: 6, close: 0, total: 220
>
> comparing that with --sync_file_range=1 --fallocate=1 --fadvise=1
> running test with: numprocs=1 filesize=429496729600 blocksize=8192 fallocate=1 sfr=1 fadvise=1
> [/srv/dev/bench/test1][14098] open: 0, fallocate: 0 write: 161, fsync: 0, close: 0, total: 161

Ah, nice.

> The run-to-run variations between the runs without cache control are
> pretty large. So this is probably not the end-all-be-all numbers. But I
> think the trends are pretty clear.

Could you be explicit about what you think those clear trends are?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company