Re: draft RFC: concept for partial, wal-based replication

On Mon, Nov 30, 2009 at 4:56 AM, Boszormenyi Zoltan <zb(at)cybertec(dot)at> wrote:
> I don't think its reasonable trying to discuss and implement this all
> in one huge patch so I propose implementing at least 1) as a seperate
> patch.

I agree with this development plan.

> Now there unforatunately are two different schools of thought how to implement transfering base data.
> 1. archive_command like transfer command taking a source path/host and target path/host
>   - very flexible (think e.g. not transferring the data for multiple slaves over the whole country)
>   - harder to setup
>   - more in style of classical wal archiving
> 2. add the capability to the WAL Streaming patch's libpq based protocol
>   - no additional configuration needed
>   - inflexible
>   - makes usage from non streaming replication is impossible
>
> I favor 1. but only lightly so.

I favor 2 ;) Because I think that it's too hard for users to set up
a transfer command. One of streaming replication's merits is that
users no longer need to specify a transfer command for log-shipping.
So users can configure and use replication without complex settings.
But, #1 would spoil this merit.

> Detail Questions:
> - How to deal with multiple transfer requests at the same time?
>  There would be a need for multiple full backup requests for
>  individual tables by several clients at once.
>  Currently pg_start_backup() isn't allowed from
>  two clients in parallel, the second one gets an error.
>  We thought that pg_start_backup() and pg_stop_backup()
>  can turn into simple reference counts. IIRC, WALs
>  are still generated and _shipped to slaves_ during
>  a full backup, they are simply not yet applied to
>  base table files. So, in this case a pg_stop_backup()
>  issued from a slave decreases refcount of the base backups
>  and the slave can simply resume applying its newly
>  received WALs to base files.

I'm not sure how. But at first multiple online-backup feature
rather than backup-shipping itself might have to be addressed.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center