Re: Walsender timeouts and large transactions

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: funny(dot)falcon(at)postgrespro(dot)ru
Cc: pgsql-hackers(at)postgresql(dot)org, pjmodos(at)pjmodos(dot)net
Subject: Re: Walsender timeouts and large transactions
Date: 2017-09-29 08:26:35
Message-ID: 20170929.172635.133003609.horiguchi.kyotaro@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello,

At Wed, 27 Sep 2017 14:28:37 +0300, Sokolov Yura <funny(dot)falcon(at)postgrespro(dot)ru> wrote in <90bb67da7131e6186b50897c4b0f0ec3(at)postgrespro(dot)ru>
> On 2017-09-12 11:28, Kyotaro HORIGUCHI wrote:
> > Hello,
> > At Wed, 06 Sep 2017 13:46:16 +0000, Yura Sokolov
> > <funny(dot)falcon(at)postgrespro(dot)ru> wrote in
> > <20170906134616(dot)18925(dot)88390(dot)pgcf(at)coridan(dot)postgresql(dot)org>
> > As the result, I think that the functions should be modified as
> > the following.
> > - Forcing slow-path if time elapses a half of a ping period is
> > right. (GetCurrentTimestamp is anyway requried.)
> > - The slow-path should not sleep waiting Latch. It should just
> > pg_usleep() for maybe 1-10ms.
> > - We should go to the fast path just after keepalive or response
> > message has been sent. In other words, the "if (now <" block
> > should be in the "for (;;)" loop. This avoids needless runs on
> > the slow-path.
> > It would be refactorable as the following.
> > prepare for the send buffer;
> > for (;;)
> > {
> > now = GetCurrentTimeStamp();
> > if (now < )...
> > {
> > fast-path
> > }
> > else
> > {
> > slow-path
> > }
> > return if finished
> > sleep for 1ms?
> > }
> > What do you think about this?
> > regards,
> > --
> > Kyotaro Horiguchi
> > NTT Open Source Software Center
>
> Good day, Petr, Kyotaro
>
> I've created failing test for issue (0001-Add-failing-test...) .
> It tests insertion of 20000 rows with 10ms wal_sender_timeout
> (it fails in WalSndWriteData on master) and then deletion of
> those rows with 1ms wal_sender_timeout (it fails in WalSndLoop).
>
> Both Peter's patch and my simplified suggestion didn't pass the
> test. I didn't checked Kyotaro's suggestion, though, cause I
> didn't understand it well.

Mmm. The test seems broken. wal_sender_timeout = 10ms with
wal_receiver_status_interval=10s immediately causes a
timeout. Avoiding the timeout is just breaking the sane code.

wal_sender_timeout = 3s and wal_receiver_status_interval=1s
effectively causes the problem with about 1000000 lines of (int)
insertion on UNIX socket connection, on my poor box.

The original complain here came from the fact that
WalSndWriteData skips processing of replies for a long time on a
fast network. However Petr's patch fixed the problem, I pointed
that just letting the function take the slow path leads to
another problem, that is, waiting for new WAL records can result
in a unwanted pause in the slow path.

Combining the solutions for the two problem is my proposal sited
above. The sentence seems in quite bad style but the attached
file is the concorete patch of that.

Any thoughts?

regards,
--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
fix_walsender_timeouts_horiguti_20170929.diff text/x-patch 4.4 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Craig Ringer 2017-09-29 08:27:58 Re: pg_prepared_xact_status
Previous Message Craig Ringer 2017-09-29 08:22:01 Re: pg_prepared_xact_status