Confused about max_standby_streaming_delay

A server running PostgreSQL 9.4 on Centos 6.8, with a live server and a hot
standby, is supporting about 20 customer organisations, each with their own
linux user and its own installation/copy of the system, talking to its own
database.

The system has a backup script which is a wrapper round pg_dump.
For most installations, it takes "a couple of minutes" to run, though
others take longer.

On the standby server, we have a directory full of cron jobs -- one for
each installation -- to invoke this backup script
Each script is run as soon as the previous one is finished. So when all
goes to plan, we get a dump of
database_a at 5 minutes past the hour,
database_b at 7 minutes past,
database_c at 9 minutes past
and so on.

H
​owever, we o​
ccasionally
​ have​
a dump die with a message
​ like​

Error message from server: ERROR: canceling statement due to
conflict with recovery

I believe this means that an update from the live server wants to change a
table that is being dumped, and so the dump is aborted.

And I've read that the answer to this is to set max_standby_streaming_delay
in postgresql94.conf.
So I've set it to "600s" -- ten minutes.

I thought this would mean that when there was a conflict with an update
from the live server, Postgres would give the dump 10 minutes "grace" in
which to finish before killing it.

Ten minutes may or may not be enough. But in a case where it isn't enough,
and the dump is abandonned, I would expect to see something like

the dump of database_a finishing at 5 minutes past the hour,
the dump of database_b
starting after the dump of database_a,
having a conflict,
being given 10 minutes to complete, and then
being abandonned
the dump of database_c starting after the dump of database_b and
finishing (say) 3 minutes later

So the dump of database_c should finish at around 18 minutes past the hour.

BUT that is not what I am seeing.
On occasions where the dump of database_b is being abandonned, the
successful dump of
database_c is timestamped less than 10 minutes after the dump of database_a

Which does not fit with the dump of database_b being given 10 minutes in
which to finish

Have I misunderstood something? Or is Postgres not actually configured the
way I think it is?

Robert.

--
Robert Inder, 0131 229 1052 / 07808 492
213
Interactive Information Ltd, 3, Lauriston Gardens, Edinburgh EH3 9HH
Registered in Scotland, Company no. SC 150689
Interactions speak louder than
words