Re: make the stats collector shutdown without writing the statsfiles if the immediate shutdown is requested.

On 2021/03/25 19:48, Fujii Masao wrote:
>
>
> On 2021/03/25 9:31, Masahiro Ikeda wrote:
>>
>>
>> On 2021/03/24 18:36, Fujii Masao wrote:
>>>
>>>
>>> On 2021/03/24 3:51, Andres Freund wrote:
>>>> Hi,
>>>>
>>>> On 2021-03-23 15:50:46 +0900, Fujii Masao wrote:
>>>>> This fact makes me wonder that if we collect the statistics about WAL
>>>>> writing
>>>>> from walreceiver as we discussed in other thread, the stats collector should
>>>>> be invoked at more earlier stage. IIUC walreceiver can be invoked before
>>>>> PMSIGNAL_BEGIN_HOT_STANDBY is sent.
>>>>
>>>> FWIW, in the shared memory stats patch the stats subsystem is
>>>> initialized early on by the startup process.
>>>
>>> This is good news!
>>
>> Fujii-san, Andres-san,
>> Thanks for your comments!
>>
>> I didn't think about the start order. From the point of view, I noticed that
>> the current source code has two other concerns.
>>
>>
>> 1. This problem is not only for the wal receiver.
>>
>> The problem which the wal receiver starts before the stats collector
>> is launched during archive recovery is not only for the the wal receiver but
>> also the checkpointer and the bgwriter. Before starting redo, the startup
>> process sends the postmaster "PMSIGNAL_RECOVERY_STARTED" signal to launch the
>> checkpointer and the bgwriter to be able to perform creating restartpoint.
>>
>> Although the socket for communication between the stats collector and the
>> other processes is made in earlier stage via pgstat_init(), I agree to make
>> the stats collector starts earlier stage is defensive. BTW, in my
>> environments(linux, net.core.rmem_default = 212992), the socket can buffer
>> almost 300 WAL stats messages. This mean that, as you said, if the redo phase
>> is too long, it can lost the messages easily.
>>
>>
>> 2. To make the stats clear in redo phase.
>>
>> The statistics can be reset after the wal receiver, the checkpointer and
>> the wal writer are started in redo phase. So, it's not enough the stats
>> collector is invoked at more earlier stage. We need to fix it.
>>
>>
>>
>> (I hope I am not missing something.)
>> Thanks to Andres-san's work([1]), the above problems will be handle in the
>> shared memory stats patch. First problem will be resolved since the stats are
>> collected in shared memory, so the stats collector process is unnecessary
>> itself. Second problem will be resolved to remove the reset code because the
>> temporary stats file won't generated, and if the permanent stats file
>> corrupted, just recreate it.
>
> Yes. So we should wait for the shared memory stats patch to be committed
> before working on walreceiver stats patch more?

Yes, I agree.

Regards,

--
Masahiro Ikeda
NTT DATA CORPORATION