Re: archive modules

On 2021/11/03 0:03, Bossart, Nathan wrote:
> On 11/1/21, 9:44 PM, "Fujii Masao" <masao(dot)fujii(at)oss(dot)nttdata(dot)com> wrote:
>> What is the main motivation of this patch? I was thinking that
>> it's for parallelizing WAL archiving. But as far as I read
>> the patch very briefly, WAL file name is still passed to
>> the archive callback function one by one.
>
> The main motivation is provide a way to archive without shelling out.
> This reduces the amount of overhead, which can improve archival rate
> significantly.

It's helpful if you share how much this approach reduces
the amount of overhead.

> It should also make it easier to archive more safely.
> For example, many of the common shell commands used for archiving
> won't fsync the data, but it isn't too hard to do so via C.

But probably we can do the same thing even by using the existing
shell interface? For example, we can implement and provide
the C program of the archive command that fsync's the file?
Users can just use it in archive_command.

> The
> current proposal doesn't introduce any extra infrastructure for
> batching or parallelism, but it is probably still possible. I would
> like to eventually add batching, but for now I'm only focused on
> introducing basic archive module support.

Understood. I agree that it's reasonable to implement them gradually.

>> Are you planning to extend this mechanism to other WAL
>> archiving-related commands like restore_command? I can imagine
>> that those who use archive library (rather than shell) would
>> like to use the same mechanism for WAL restore.
>
> I would like to do this eventually, but my current proposal is limited
> to archive_command.

Understood.

>> I think that it's worth adding this module into core
>> rather than handling it as test module. It provides very basic
>> WAL archiving feature, but (I guess) it's enough for some users.
>
> Do you think it should go into contrib?

Yes, at least for me..

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION