Re: autonomous transactions

On 8 Sep. 2016 1:38 pm, "Andres Freund" <andres(at)anarazel(dot)de> wrote:
>
> I kind of dislike this approach for a different reason than already
> mentioned in this thread: Namely it's not re-usable for implementing
> streaming logical replication of large transaction, i.e. allow to decode
> & apply un-committed changes. The issue there is that one really needs
> to have multiple transactions active in the same connection, which this
> approach does not allow.

I've been thinking about this recently with an eye to handling the majority
of transactions on a typical system that neither perform DDL nor are
concurrent with it.

The following might be fairly nonsensical if I've misunderstood the problem
as I've not had a chance to more than glance at it, but:

I wonder if some kind of speculative concurrent decoding+output is possible
without being able to have multiple xacts on a backend. We could use a
shared xact and snapshot for all concurrent xacts. If any of them do
anything that requires invalidations etc we dump the speculatively
processed ones and fall back to reorder buffering and serial output at
commit time until we pass the xact that caused an invalidation and don't
have more pending. Add a callback to the output plugin to tell it that
speculative decoding of an xact has been aborted.

Or maybe even just put that speculstive decoding on hold and pick up where
we left off partway through the reorder buffer when we get around to
processing it serially. That way we don't have to discard work already
done. More usefully we could avoid having to enqueue stuff into the reorder
buffer just in case we have to switch to fallback serial decoding.

Failing that:

Since logical decoding only needs read only xacts that roll back, it only
needs multiple backend private virtualxacts. They don't need
SERIALIZABLE/SSI which I think (?) means other backends don't need to be
able to wait on them. That seems simpler than what autonomous xacts would
need since there we need them exposed in PgXact, waitable-on for txid
locks, etc. Right? In the same way that historical snapshots are cut-down
maybe logical decoding xacts can be too.

I suspect that waiting for full in-process multiple xact support to do
interleaved decoding/replay will mean a long wait indeed.