Re: [BUGS] BUG #7534: walreceiver takes long time to detect n/w breakdown

On Thursday, November 08, 2012 10:42 PM Fujii Masao wrote:
> On Thu, Nov 8, 2012 at 5:53 PM, Amit Kapila <amit(dot)kapila(at)huawei(dot)com>
> wrote:
> > On Thursday, November 08, 2012 2:04 PM Heikki Linnakangas wrote:
> >> On 19.10.2012 14:42, Amit kapila wrote:
> >> > On Thursday, October 18, 2012 8:49 PM Fujii Masao wrote:
> >> >> Before implementing the timeout parameter, I think that it's
> better
> >> to change
> >> >> both pg_basebackup background process and pg_receivexlog so that
> they
> >> >> send back the reply message immediately when they receive the
> >> keepalive
> >> >> message requesting the reply. Currently, they always ignore such
> >> keepalive
> >> >> message, so status interval parameter (-s) in them always must be
> set
> >> to
> >> >> the value less than replication timeout. We can avoid this
> >> troublesome
> >> >> parameter setting by introducing the same logic of walreceiver
> into
> >> both
> >> >> pg_basebackup background process and pg_receivexlog.
> >> >
> >> > Please find the patch attached to address the modification
> mentioned
> >> by you (send immediate reply for keepalive).
> >> > Both basebackup and pg_receivexlog uses the same function
> >> ReceiveXLogStream, so single change for both will address the issue.
> >>
> >> Thanks, committed this one after shuffling it around the changes I
> >> committed yesterday. I also updated the docs to not claim that -s
> option
> >> is required to avoid timeout disconnects anymore.
> >
> > Thank you.
> > However I think still the issue will not be completely solved.
> > pg_basebackup/pg_receivexlog can still take long time to
> > detect network break as they don't have timeout concept. To do that I
> have
> > sent one proposal which is mentioned at end of mail chain:
> > http://archives.postgresql.org/message-
> id/6C0B27F7206C9E4CA54AE035729E9C3828
> > 53BBED(at)szxeml509-mbs
> >
> > Do you think there is any need to introduce such mechanism in
> > pg_basebackup/pg_receivexlog?
>
> Are you planning to introduce the timeout mechanism in pg_basebackup
> main process? Or background process? It's useful to implement both.

By background process, you mean ReceiveXlogStream?
For both.

I think for background process, it can be done in a way similar to what we
have done for walreceiver.
But I have some doubts for how to do for main process:

Logic similar to walreceiver can not be used incase network goes down during
getting other database file from server.
The reason for the same is to receive the data files PQgetCopyData() is
called in synchronous mode, so it keeps waiting for infinite time till it
gets some data.
In order to solve this issue, I can think of following options:
1. Making this call also asynchronous (but now sure about impact of this).
2. In function pqWait, instead of passing hard-code value -1 (i.e. infinite
wait), we can send some finite time. This time can be received as command
line argument
from respective utility and set the same in PGconn structure.
In order to have timeout value in PGconn, we can have:
a. Add new parameter in PGconn to indicate the receive timeout.
b. Use the existing parameter connect_timeout for receive timeout
also but this may lead to confusion.
3. Any other better option?

Apart from above issue, there is possibility that if during connect time
network goes down, then it might hang, because connect_timeout by default
will be NULL and connectDBComplete will start waiting inifinitely for
connection to become successful.
So shall we have command line argument separately for this also or any other
way as you suugest.

> BTW, IIRC the walsender has no timeout mechanism during sending
> backup data to pg_basebackup. So it's also useful to implement the
> timeout mechanism for the walsender during backup.

Yes, its useful, but for walsender the main problem is that it uses blocking
send call to send the data.
I have tried using tcp_keepalive settings, but the send call doesn't comeout
incase of network break.
The only way I could get it out is:
change in the corresponding file /proc/sys/net/ipv4/tcp_retries2 by using
the command
echo "8" > /proc/sys/net/ipv4/tcp_retries2
As per recommendation, its value should be at-least 8 (equivalent to 100
sec)

Do you have any idea, how it can be achieved?

With Regards,
Amit Kapila.