Re: [FEATURE REQUEST] Streaming Onlinebackup (MaybeOFFTOPIC)

>>> On Fri, Sep 28, 2007 at 9:38 AM, in message
<46FCCB89(dot)EE98(dot)0025(dot)0(at)wicourts(dot)gov>, "Kevin Grittner"
<Kevin(dot)Grittner(at)wicourts(dot)gov> wrote:
>>>> On Fri, Sep 28, 2007 at 5:53 AM, in message
> <E1539E0ED7043848906A8FF995BDA57902685F43(at)m0143(dot)s-mxs(dot)net>, "Zeugswetter
> Andreas ADI SD" <Andreas(dot)Zeugswetter(at)s-itsolutions(dot)at> wrote:

>> archive_command=dd if=%p of=/backup/WAL/%f bs=1 count=%b

> I think
> your example would need to cat the dd you showed with one which drew
> from /dev/zero. I'll run a few tests with full and nearly empty files
> using hand-generated values and see how the performance of this in
> front of gzip compares to the filter.

After Tom's email, this is pretty academic; but here are the results
for our "best case" example:

pg_clearxlogtail | gzip:
real 0m0.132s
user 0m0.119s
sys 0m0.024s

(dd if=00000001000000040000001A bs=1 count=132 ; dd if=/dev/zero bs=1 count=16777084) | gzip > ../kjgtest2/00000001000000040000001A.2.gz
132+0 records in
132+0 records out
16777084+0 records in
16777084+0 records out

real 0m19.243s
user 0m3.211s
sys 0m27.135s

That's a lot worse. I switched the bs and count:

(dd if=00000001000000040000001A bs=132 count=1 ; dd if=/dev/zero bs=16777084 count=1) | gzip > ../kjgtest2/00000001000000040000001A.3.gz
1+0 records in
1+0 records out
1+0 records in
1+0 records out

real 0m0.196s
user 0m0.173s
sys 0m0.025s

The filter code still wins.

The "worst case" example:

pg_clearxlogtail | gzip:
real 0m1.073s
user 0m1.018s
sys 0m0.063s
gz size: 4554307

ADAMS-PG:/var/pgsql/data/kjgtest # time dd if=0000000100000003000000F0 bs=16777216 count=1 | gzip > ../kjgtest2/0000000100000003000000F0.3.gz
1+0 records in
1+0 records out

Marginal improvement.

real 0m1.001s
user 0m0.923s
sys 0m0.081s

ADAMS-PG:/var/pgsql/data/kjgtest # time cat 0000000100000003000000F0 | gzip > ../kjgtest2/0000000100000003000000F0.4.gz

real 0m1.109s
user 0m1.055s
sys 0m0.062s

Not quite as good. Since the archiver process can't actually deliver
this number in a lightweight manner, all it goes to show is that the
filter code compares reasonably well in performance with dd and cat.

-Kevin