Re: Transactions involving multiple postgres foreign servers

On Thu, Oct 26, 2017 at 2:36 PM, Ashutosh Bapat
<ashutosh(dot)bapat(at)enterprisedb(dot)com> wrote:
> On Wed, Oct 25, 2017 at 3:15 AM, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>>
>> Foreign Transaction Resolver
>> ======================
>> I introduced a new background worker called "foreign transaction
>> resolver" which is responsible for resolving the transaction prepared
>> on foreign servers. The foreign transaction resolver process is
>> launched by backend processes when commit/rollback transaction. And it
>> periodically resolves the queued transactions on a database as long as
>> the queue is not empty. If the queue has been empty for the certain
>> time specified by foreign_transaction_resolver_time GUC parameter, it
>> exits. It means that the backend doesn't launch a new resolver process
>> if the resolver process is already working. In this case, the backend
>> process just adds the entry to the queue on shared memory and wake it
>> up. The maximum number of resolver process we can launch is controlled
>> by max_foreign_transaction_resolvers. So we recommends to set larger
>> max_foreign_transaction_resolvers value than the number of databases.
>> The resolver process also tries to resolve dangling transaction as
>> well in a cycle.
>>
>> Processing Sequence
>> =================
>> I've changed the processing sequence of resolving foreign transaction
>> so that the second phase of two-phase commit protocol (COMMIT/ROLLBACK
>> prepared) is executed by a resolver process, not by backend process.
>> The basic processing sequence is following;
>>
>> * Backend process
>> 1. In pre-commit phase, the backend process saves fdwxact entries, and
>> then prepares transaction on all foreign servers that can execute
>> two-phase commit protocol.
>> 2. Local commit.
>> 3. Enqueue itself to the shmem queue and change its status to WAITING
>> 4. launch or wakeup a resolver process and wait
>>
>> * Resolver process
>> 1. Dequeue the waiting process from shmem qeue
>> 2. Collect the fdwxact entries that are associated with the waiting process.
>> 3. Resolve foreign transactoins
>> 4. Release the waiting process
>
> Why do we want the the backend to linger behind, once it has added its
> foreign transaction entries in the shared memory and informed resolver
> about it? The foreign connections may take their own time and even
> after that there is no guarantee that the foreign transactions will be
> resolved in case the foreign server is not available. So, why to make
> the backend wait?

Because I don't want to break the current user semantics. that is,
currently it's guaranteed that the subsequent reads can see the
committed result of previous writes even if the previous transactions
were distributed transactions. And it's ensured by writer side. If we
can make the reader side ensure it, the backend process don't need to
wait for the resolver process.

The waiting backend process are released by resolver process after the
resolver process tried to resolve foreign transactions. Even if
resolver process failed to either connect to foreign server or to
resolve foreign transaction the backend process will be released and
the foreign transactions are leaved as dangling transaction in that
case, which are processed later. Also if resolver process takes a long
time to resolve foreign transactions for whatever reason the user can
cancel it by Ctl-c anytime.

>>
>> 5. Wake up and restart
>>
>> This is still under the design phase and I'm sure that there is room
>> for improvement and consider more sensitive behaviour but I'd like to
>> share the current status of the patch. The patch includes regression
>> tests but not includes fully documentation.
>
> Any background worker, backend should be child of the postmaster, so
> we should not let a backend start a resolver process. It should be the
> job of the postmaster.
>

Of course I won't. I used the term of "the backend process launches
the resolver process" for explaining easier. Sorry for confusing you.
The backend process calls RegisterDynamicBackgroundWorker() function
to launch a resolver process, so they are launched by postmaster.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center