Skip site navigation (1) Skip section navigation (2)

pg_upgrade with parallel tablespace copying

From: Bruce Momjian <bruce(at)momjian(dot)us>
To: PostgreSQL-development <pgsql-hackers(at)postgreSQL(dot)org>
Subject: pg_upgrade with parallel tablespace copying
Date: 2013-01-08 03:51:21
Message-ID: 20130108035121.GA12697@momjian.us (view raw or flat)
Thread:
Lists: pgsql-hackers
Pg_upgrade by default (without --link) copies heap/index files from the
old to new cluster.  This patch implements parallel heap/index file
copying in pg_upgrade using the --jobs option.  It uses the same
infrastructure used for pg_upgrade parallel dump/restore.  Here are the
performance results:

	           --- seconds ---
	   GB        git    patched
	    2       62.09    63.75
	    4       95.93   107.22
	    8      194.96   195.29
	   16      494.38   348.93
	   32      983.28   644.23
	   64     2227.73  1244.08
	  128     4735.83  2547.09

Because of the kernel cache, you only see a big win when the amount of
copy data exceeds the kernel cache.  For testing, I used a 24GB, 16-core
machine with two magnetic disks with one tablespace on each.  Using more
tablespaces would yield larger improvements.  My test script is
attached.  

I consider this patch ready for application.  This is the last
pg_upgrade performance improvement idea I am considering.

-- 
  Bruce Momjian  <bruce(at)momjian(dot)us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Attachment: test_many_tables
Description: text/plain (1.6 KB)
Attachment: tablespace.diff
Description: text/x-diff (25.4 KB)

Responses

pgsql-hackers by date

Next:From: Pavan DeolaseeDate: 2013-01-08 06:17:17
Subject: Re: Set visibility map bit after HOT prune
Previous:From: Peter EisentrautDate: 2013-01-08 03:16:56
Subject: Re: [PATCH] Patch to fix missing libecpg_compat.lib and libpgtypes.lib.

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group