Re: block-level incremental backup

> 21 апр. 2019 г., в 1:13, Robert Haas <robertmhaas(at)gmail(dot)com> написал(а):
>
> On Sat, Apr 20, 2019 at 12:44 PM Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
>> Incremental backup of 1Tb DB made with distance of few minutes (small change set) is few Gbs. All of this size is made of FSM (no LSN) and VM (hard to use LSN).
>> Sure, this overhead size is fine if we make daily backup. But at some frequency of backups it will be too much.
>
> It seems like if the backups are only a few minutes apart, PITR might
> be a better choice than super-frequent incremental backups. What do
> you think about that?
PITR is painfully slow on heavily loaded clusters. I observed restorations when 5 seconds of WAL were restored in 4 seconds. Backup was only few hours past primary node, but could catch up only at night.
And during this process only one of 56 cpu cores was used. And SSD RAID throughput was not 100% utilized.

Block level delta backups can be restored very efficiently: if we restore from newest to past steps, we write no more than cluster size at last backup.

>> I think that problem of incrementing FSM and VM is too distant now.
>> But if I had to implement it right now I'd choose following way: do not backup FSM and VM, recreate it during restore. Looks like it is possible, but too much AM-specific.
>
> Interesting idea - that's worth some more thought.

Core routines to recreate VM and FSM would be cool :) But this need to be done without extra IO, not an easy trick.

>> Here's 53 mentions of "parallel backup". I want to note that there may be parallel read from disk and parallel network transmission. Things between these two are neglectable and can be single-threaded. From my POV, it's not about threads, it's about saturated IO controllers.
>> Also I think parallel restore matters more than parallel backup. Backups themself can be slow, on many clusters we even throttle disk IO. But users may want parallel backup to catch-up standby.
>
> I'm not sure I entirely understand your point here -- are you saying
> that parallel backup is important, or that it's not important, or
> something in between? Do you think it's more or less important than
> incremental backup?
I think that there is no such thing as parallel backup. Backup creation is composite process of many subprocesses.

In my experience, parallel network transmission is cool and very important, it makes upload 3 times faster. But my experience is limited to cloud storages. Would this hold if storage backend is local FS? I have no idea.
Parallel reading from disk has the same effect. Compression and encryption can be single threaded, I think it will not be bottleneck (unless one uses lzma's neighborhood on Pareto frontier).

For me, I think the most important thing is incremental backups (with parallel steps merge) and then parallel backup.
But there is huge fraction of users, who can benefit from parallel backup and do not need incremental backup at all.

Best regards, Andrey Borodin.