Re: Error "initial slot snapshot too large" in create replication slot

On Thu, Jan 11, 2024 at 9:47 PM Kyotaro Horiguchi
<horikyota(dot)ntt(at)gmail(dot)com> wrote:
> I understand. So, summirizing the current state briefly, I believe it
> as follows:
>
> a. snapbuild lacks a means to differentiate between top and sub xids
> during snapshot building.
>
> b. Abusing takenDuringRecovery could lead to potential issues, so it
> has been rejected.
>
> c. Dynamic sizing of xip is likely to have a significant impact on
> performance (as mentioned in the comments of GetSnapshotData).
>
> d. (new!) Adding a third storing method is not favored.
>
> As a method to satisfy these prerequisites, I think one direction
> could be to split takenDuringRecovery into flags indicating the
> storage method and creation timing. I present this as a last-ditch
> effort. It's a rough proposal, and further validation should be
> necessary. If this direction is also not acceptable, I'll proceed with
> removing the CF entry.

I think that the idea of evolving takenDuringRecovery could
potentially work for this problem and maybe for some other things as
well. I remember studying that flag before and coming to the
conclusion that it had some pretty specific and surprising semantics
that weren't obvious from the name -- I don't remember the details --
and so I think improving it could be useful. Also, I think that
multiple storage methods could be more palatable if there were a clear
way to distinguish which one was in use and good comments explaining
the distinction in relevant places.

However, I wonder whether this whole area is in need of a bigger
rethink. There seem to be a number of situations in which the split
into xip and subxip arrays is not very convenient, and also some
situations where it's quite important. Sometimes we want to record
what's committed, and sometimes what isn't. It's all a bit messy and
inconsistent. The necessity of limiting snapshot size is annoying,
too. I have no real idea what can be done about all of this, but what
strikes me is that the current system has grown up incrementally: we
started with a data structure designed for the original use case, and
now by gradually adding new use cases things have gotten complicated.
If we were designing things over from scratch, maybe we'd do it
differently and end up with something less messy. And maybe someone
can imagine a redesign that is likewise less messy.

But on the other hand, maybe not. Perhaps we can't really do any
better than what we have. Then the question becomes whether this case
is important enough to justify additional code complexity. I don't
think I've personally seen users run into this problem so I have no
special reason to think that it's important, but if it's causing
issues for other people then maybe it is.

--
Robert Haas
EDB: http://www.enterprisedb.com