Re: Logical Decoding and HeapTupleSatisfiesVacuum assumptions

On 29 January 2018 at 07:15, Nikhil Sontakke <nikhils(at)2ndquadrant(dot)com> wrote:

>> Having this as responsibility of plugin sounds interesting. It certainly
>> narrows the scope for which we need to solve the abort issue. For 2PC
>> that may be okay as we need to somehow interact with transaction manager
>> as Simon noted. I am not sure if this helps streaming use-case though as
>> there is not going to be any external transaction management involved there.

I think we should recognize that the two cases are different.

2PC decoding patch is looking at what happens AFTER a PREPARE has
occurred on a transaction.

Streaming looks to have streaming occur right in the middle of a
transaction, so it has other concerns and likely other solutions as a
result.

>> In any case all this interlocking could potentially be made less
>> impact-full by only doing it when we know the transaction did catalog
>> changes prior to currently decoded change (which we do during decoding)
>> since that's the only time we are interested in if it aborted or not.
>>
>> This all leads me to another idea. What if logical decoding provided API
>> for "locking/unlocking" the currently decoded transaction against abort.
>> This function would then be called by both decoding and output plugin
>> before any catalog read. The function can be smart enough to be NOOP if
>> the transaction is not running (ie we are not doing 2PC decoding or
>> streaming) or when the transaction didn't do any catalog modifications
>> (we already have that info easily accessible as bool).
>>
>> That would mean we'd never do any kind of heavy locking for prolonged
>> periods of time (ie network calls) but only during catalog access and
>> only when needed. It would also solve this for both 2PC and streaming
>> and it would be easy to use by plugin authors. Just document that some
>> call should be done before catalog access when in output plugin, can
>> even be Asserted that the call was done probably.
>>
>> Thoughts?
>>
>
> Yeah, this might work. We already have SET_LOCKTAG_TRANSACTION() via
> which every transaction takes an exclusive lock on its own
> transactionid when it starts, for example. We ideally want a single
> solution to handle 2PC and ongoing (streaming) transactions. We could
> introduce a new SET_LOCKTAG_LOGICALTRANSACTION(). The logical decoding
> process could take a SHARED lock on this, check if the XID is still ok
> to decode, read the catalog and unlock. Abort/Commit transaction
> processing could take this in EXCLUSIVE mode.
>
> As mentioned above, the plugin API which takes this lock will be smart
> enough to be a NOOP if the transaction is not running (i.e we are not
> doing 2PC decoding or streaming) or when the transaction didn't do any
> catalog modifications.

I think it is enough to say that ROLLBACK PREPARED cannot abort a
transaction while decoding is taking place, if the plugin decides it
wishes to prevent that. So the plugin API must have a
pre-rollback-prepared call to allow the plugin to decide how to handle
that (Is that what you meant by the pre-abort hook?). That call will
just return when and if it is OK to abort. Any waiting or throwing of
ERRORs can occur inside the plugin.

The plugin can provide a function to allow an abort that type of abort
if it wishes, but no new core functionality required for that. If
needed, we would wait on a latch the plugin provides for that.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services