Re: WIP/PoC for parallel backup

From: Asif Rehman <asifr(dot)rehman(at)gmail(dot)com>
To: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Cc: Kashif Zeeshan <kashif(dot)zeeshan(at)enterprisedb(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, Rajkumar Raghuwanshi <rajkumar(dot)raghuwanshi(at)enterprisedb(dot)com>, Jeevan Chalke <jeevan(dot)chalke(at)enterprisedb(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: WIP/PoC for parallel backup
Date: 2020-04-21 07:29:47
Message-ID: CADM=JehhQpwgs9k-N7tdokAjrTH-RYaoBRSrGhxCtTdAWj6+xQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

I did some tests a while back, and here are the results. The tests were
done to simulate
a live database environment using pgbench.

machine configuration used for this test:
Instance Type: t2.xlarge
Volume Type : io1
Memory (MiB) : 16384
vCPU # : 4
Architecture : X86_64
IOP : 16000
Database Size (GB) : 102

The setup consist of 3 machines.
- one for database instances
- one for pg_basebackup client and
- one for pgbench with some parallel workers, simulating SELECT loads.

basebackup | 4 workers | 8 Workers | 16
workers
Backup Duration(Min): 69.25 | 20.44 | 19.86 | 20.15
(pgbench running with 50 parallel client simulating SELECT load)

Backup Duration(Min): 154.75 | 49.28 | 45.27 | 20.35
(pgbench running with 100 parallel client simulating SELECT load)

On Tue, Apr 21, 2020 at 9:27 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:

> On Tue, Apr 14, 2020 at 8:07 PM Asif Rehman <asifr(dot)rehman(at)gmail(dot)com>
> wrote:
>
>>
>> I forgot to make a check for no-manifest. Fixed. Attached is the updated
>> patch.
>>
>>
> Have we done any performance testing with this patch to see the benefits?
> If so, can you point me to the results? If not, then can we perform some
> tests on large backups to see the benefits of this patch/idea?
>
> --
> With Regards,
> Amit Kapila.
> EnterpriseDB: http://www.enterprisedb.com
>

--
--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Kyotaro Horiguchi 2020-04-21 07:58:22 Re: Remove non-fast promotion Re: Should we remove a fallback promotion? take 2
Previous Message Amit Kapila 2020-04-21 07:21:06 Re: PG compilation error with Visual Studio 2015/2017/2019