Re: PostgreSQL logical decoder output plugin - unchanged toast data

Hi,

Hi,

On 2018-08-31 15:36:26 +0200, Georgy Buranov wrote:
> I am not specialist in Postgres at all, so maybe I miss some very
> basic point. In the plugin, I want to always get all the values (even
> those that are unchanged toast data)

> When I try to get the datum that is internal on disk (here is code)
>
> ```
> struct varlena *s = (struct varlena *)DatumGetPointer(origval);
> struct varlena * ret = heap_tuple_untoast_attr(s);
> Datum result = PointerGetDatum(PG_DETOAST_DATUM(ret));
> ```
> it fails with no known snapshots error (from heap_tuple_untoast_attr).
> My question is why is it that.

Yes, that's not possible in general. On-disk toasted data for tuples
from the WAL are not guaranteed in any way to be retained. If that
weren't the case database tables would bloat while logical replication
is behind, and the sequential reads (i.e. fast) reads of logical
decoding would turn into random IO.

You can however alter the replication identity of tables to FULL. Then
the "old" tuple in change callbacks will have the full old tuple. But
that will increase the size of the WAL stream obviously.

> So, even theoretically it is not possible to the the varlena on disk
> from logical replication plugin?

Correct.

Greetings,

Andres Freund