Re: Using streaming replication as log archiving

On Wed, Sep 29, 2010 at 05:40, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Tue, Sep 28, 2010 at 5:23 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> When I ran that, the size of the WAL file in inprogress directory
>>> became more than 16MB. Obviously something isn't right.
>>
>> Wow, that's weird. I'm unable to reproduce that here - can you try to
>> figure out why that happened?
>
> Sorry, I overlooked the single-digit figure in the result of "ls -l".

Aha, that explains it.

> To be exact, the size of the WAL file in inprogress directory can be
> less than 16MB. Here is the result of "ls -l inprogress".
>
> $ ls -l inprogress/
> total 1724
> -rw-rw-r-- 1 postgres postgres 1757352 Sep 29 12:03 000000010000000000000003
>
> This also would be problem since the WAL file smaller than 16MB cannot
> be used for recovery. I think that pg_streamrecv should create 16MB
> file with zero at first, and write the received WAL records in that, as
> walreceiver does.

It's actually intentional. If we create a file at first, there is no
way to figure out exactly how far through a partial segment we are
without parsing the details of the log. This is useful both for the
admin (who can look at the directory and watch the file grow) and the
tool itself (to know when the .save file can be rotated away, when
recovering from a partial segment receive).

My idea was to just have the admin pad the file when it's time to do
the restore. I could perhaps even add an option to the tool to do it -
the idea being it's a manual step still.

Do you have another suggestion for how to provide those two things?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/