Re: Reliable WAL file shipping over unreliable network

So, I decided to run a test.

It turns out we are both wrong.

My test script is as follows:

#======================================================================
#!/bin/bash

# Remote machine
MACH=dianne.skoll.ca

# Create a 10MB pseudo-random file
echo "`date` Creating 10MB file"
FILE=pseudo-random
dd if=/dev/urandom bs=1k count=10k of=$FILE 2>/dev/null

# Rsync it over at 500kB per second in the background
echo "`date` Starting rsync"
rsync -c --verbose --bwlimit=500k $FILE $MACH:/tmp &
RSYNC_PID=$!

echo "`date` rsync PID=$RSYNC_PID; sleeping for 3 seconds"
# Sleep for 3 seconds
sleep 3

echo "`date` Stopping rsync"
# Stop the rsync
kill -STOP $RSYNC_PID

echo "`date` Changing source file"
# Change the source file
dd if=/dev/urandom bs=1k count=10k of=$FILE 2>/dev/null

echo "`date` Continuing rsync"
# Restart the rsync and bring it into the foreground
kill -CONT $RSYNC_PID
wait

echo "`date` Rsync done; comparing files"

ssh $MACH "cd /tmp && sha1sum $FILE" | sha1sum -c -
#======================================================================

Running it produces this output:

Wed Feb 28 14:14:32 EST 2018 Creating 10MB file
Wed Feb 28 14:14:32 EST 2018 Starting rsync
Wed Feb 28 14:14:32 EST 2018 rsync PID=24456; sleeping for 3 seconds
pseudo-random
Wed Feb 28 14:14:35 EST 2018 Stopping rsync
Wed Feb 28 14:14:35 EST 2018 Changing source file
Wed Feb 28 14:14:35 EST 2018 Continuing rsync

sent 10,488,419 bytes received 19,505 bytes 488,740.65 bytes/sec
total size is 10,485,760 speedup is 1.00
Wed Feb 28 14:14:53 EST 2018 Rsync done; comparing files
pseudo-random: FAILED
sha1sum: WARNING: 1 computed checksum did NOT match

So even *with* the -c flag, rsync didn't notice the file had been changed
while it was running. Interesting.

Regards,

Dianne.