Re: Methods to quickly spin up copies of an existing databases

Pre-copying is not really an option since we could potentially need 1-X
instances so it needs to be scalable. XFS also allows for "cp --reflink"
which I could do on a PGDATA directory and then change the port number.
That's probably the method I'll try first.

We do use barman, but again a barman recover operation is pretty much just
an rsync.

I will first try XFS, then ZFS, and if those don't work, I'll look into a
SAN option.

On Fri, Mar 1, 2019 at 4:29 PM Bruce Klein <brucek(at)gmail(dot)com> wrote:

> Apologies for the low tech suggestion, but if this really is a clone of a
> previously existing template, could the clone operation just be done ahead
> of time? I.e., have the build server keep X copies ready for use and
> generate additional copies as those are consumed, so that the cloning is no
> longer on the critical path?
>
> On Fri, Mar 1, 2019 at 11:09 AM Jerry Sievers <gsievers19(at)comcast(dot)net>
> wrote:
>
>> Kenneth Marshall <ktm(at)rice(dot)edu> writes:
>>
>> > On Fri, Mar 01, 2019 at 11:57:30AM -0800, Kevin Wilkinson wrote:
>> >
>> >> if you are able/willing to use ZFS (rather than ext4, xfs, ...) to
>> >> store your database, then it might work for you. ZFS is
>> >> copy-on-write so it can very quickly clone a database.
>> >>
>> >> kevin
>> >
>> > Hi Arjun
>> >
>> > Redhat 7 does have LVM snapshots that does something similar. Kevin is
>> > correct, COW is the secret.
>>
>> Going a bit further...
>>
>> Any sort of storage backend that can support *atomic* snapshots across
>> *all* volumes (in case multiple tablespaces ar involved), can be used to
>> permit $instantaneous cloning where instantaneous relates to the actual
>> snapshot time and crash recovery.
>>
>> Inability to make *atomic* snaps but perhaps seperate snaps very
>> quickly, combined with PITR can result in clones of high-churn systems
>> sized in TBs (as in our use case) to be provisioned in about 1 minute.
>>
>> Nothing but the most trivial system can be cloned rapidly and perhaps
>> any number of times in succession without employment of
>> thin-provisioning, copy-on-write (as mentioned already), etc.
>>
>> Virtual copy is more and more compelling as physical
>> size, or more precisely, *physical* copy time grow.
>>
>> HTH
>>
>>
>>
>> >
>> > Regards,
>> > Ken
>> >
>> >
>>
>> --
>> Jerry Sievers
>> Postgres DBA/Development Consulting
>> e: postgres(dot)consulting(at)comcast(dot)net
>>
>>