Re: Testing of parallel restore with current snapshot

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Tom Lane wrote:
>> Based on this thought, what seems to make sense as a quick-and-dirty
>> answer is to make sure that items get scheduled in the same order they
>> came free from dependency restrictions. I don't recall whether that
>> is true at the moment, or how hard it might be to make it true if it
>> isn't already.

> AIUI, pg_dump sorts items by <object-type, schema, objectname> and then
> does a topological sort to permute this order to reflect dependencies.
> This is the TOC order parallel restore starts with (unless the order is
> mucked with by the user via the --use-list option). Each time it needs
> to schedule an item from the list, it chooses the first one yet to run
> that meets both these criteria:

> * all its dependencies have already been restored
> * it has no locking conflicts with a currently running item.

Hmm, the locking-conflicts point puts a bit of a hole in my thinking,
because that's non-monotonic --- an item that was ready to run a moment
ago might no longer be ready to run after you start some other task that
has a lock conflict with it.

Anyway, the idea I'd had was to use two lists: a list of jobs that are
still blocked by dependencies, and a list of jobs that have no remaining
dependencies. Whenever a job finishes, run through the former list and
transfer any no-longer-blocked items to the end of the latter list.
Then the selection rule is to run the first element of the latter list
that hasn't got any locking conflicts. This would tend to preserve the
behavior of executing things in the order they come free, though it
wouldn't be perfect because of the locking issue. It would be easy
enough to code up, and probably cleaner/faster than what's there now
(which has a weird hack to avoid O(N^2) behavior due to examining the
*entire* TOC list every time). I'm unconvinced though that it would
make any meaningful difference, and I'm not in a position to do serious
testing. Is anyone interested enough to try it if I code it?

regards, tom lane