Re: pg_dump directory archive format / parallel pg_dump

On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro
<itagaki(dot)takahiro(at)gmail(dot)com> wrote:
> I think we have 2 important technical issues here:
>  * The consistency is not perfect. Each transaction is started
>   with small delays in step 1, but we cannot guarantee no other
>   transaction between them.

This is exactly where the patch for synchronized snapshot comes into
the game. See https://commitfest.postgresql.org/action/patch_view?id=480

>  * Can we inherit connections to child processes with fork() ?
>   Moreover, we also need to pass running transactions to children.
>   I wonder libpq is designed for such usage.

As far as I know you can inherit sockets to a child program, as long
as you make sure that after the fork only one, father or child, uses
the socket, the other one should close it. But this wouldn't be a
matter with the above mentioned patch anyway.

> It might be better to remove Windows-specific codes from the first try.
> I doubt Windows message queue is the best API in such console-based
> application. I hope we could use the same implementation for all
> platforms for inter-process/thread communication.

Windows doesn't support pipes, but offers the message queues to
exchange messages. Parallel pg_dump only exchanges messages in the
form of "DUMP 39209" or "RESTORE OK 48 23 93", it doesn't exchange any
large chunks of binary data, just these small textual messages. The
messages also stay within the same process, they are just sent between
the different threads. The windows part worked just fine when I tested
it last time. Do you have any other technology in mind that you think
is better suited?

Joachim