Re: [HACKERS] WAL logging problem in 9.4.3?

Hello.

At Sun, 8 Dec 2019 10:09:51 -0800, Noah Misch <noah(at)leadboat(dot)com> wrote in
> I reviewed your latest code, and it's nearly complete. mdimmedsync() syncs
> only "active segments" (as defined in md.c), but smgrDoPendingSyncs() must
> sync active and inactive segments. This matters when mdtruncate() truncated
> the relation after the last checkpoint, causing active segments to become
> inactive. In such cases, syncs of the inactive segments will have been queued
> for execution during the next checkpoint. Since we skipped the
> XLOG_SMGR_TRUNCATE record, we must complete those syncs before commit. Let's

Got it! You're so great. Thanks.

> just modify smgrimmedsync() to always sync active and inactive segments;
> that's fine to do in other smgrimmedsync() callers, even though they operate
> on relations that can't have inactive segments.

Agreed and done that way. Even it's not harmful to leave inactive
segments open but I choosed to close them after sync. As being
mentioned in the added comment in the function, inactive segments may
not be closed if error happened on file sync. md works properly even
in the case (as the file comment says) and, anyway, mdnblocks leaves
the first inactive segment open if there's no partial segment.

I don't understand why mdclose checks for (v->mdfd_vfd >= 0) of open
segment but anyway mdimmedsync is believing that that won't happen and
I follow the assumption. (I suspect that the if condition in mdclose
should be an assertion..)

> On Tue, Dec 03, 2019 at 08:51:46PM +0900, Kyotaro Horiguchi wrote:
> > At Thu, 28 Nov 2019 17:23:19 -0500, Noah Misch <noah(at)leadboat(dot)com> wrote in
> > > On Thu, Nov 28, 2019 at 09:35:08PM +0900, Kyotaro Horiguchi wrote:
> > > > I measured the performance with the latest patch set.
> > > >
> > > > > 1. Determine $DDL_COUNT, a number of DDL transactions that take about one
> > > > > minute when done via syncs.
> > > > > 2. Start "pgbench -rP1 --progress-timestamp -T180 -c10 -j10".
> > > > > 3. Wait 10s.
> > > > > 4. Start one DDL backend that runs $DDL_COUNT transactions.
> > > > > 5. Save DDL start timestamp, DDL end timestamp, and pgbench output.
>
> > wal_buffers | 512
>
> This value (4 MiB) is lower than a tuned production system would have. In
> future benchmarks (if any) use wal_buffers=2048 (16 MiB).

Yeah, only 0.5GB of shared_buffers makes the default value of
wal_buffers reach to the heaven. I think I can take numbers on that
condition. (I doubt that it's meaningful if I increase only
wal_buffers manually.)

Anyway the default value ought to be defined based on the default
configuration.

In the attached patch, I merged all pieces in the previous version and
the change made this time (only md.c is changed this time).

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center