Re: Remaining beta blockers

On 2013-04-30 08:35:32 -0700, Kevin Grittner wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > On 2013-04-30 07:33:05 -0700, Kevin Grittner wrote:
> >> Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
>
> >>> 2) Since we don't have a metapage to represent scannability in 9.3
> >>>    we cannot easily use one in 9.4+ without pg_upgrade emptying all
> >>>    matviews unless we only rely on the catalogs which we currently
> >>>    cannot.
> >
> >> I am not following this argument at all.  Doesn't pg_upgrade use
> >> pg_dump to create the tables and matviews WITH NO DATA and take
> >> later action for ones which are populated in the source?  It would
> >> not be hard at all to move to a new release which used a different
> >> technique for tracking populated tables by changing what pg_dump
> >> does for populated tables with the switch pg_upgrade uses.
> >
> > What I am thinking about is a 100GB materialized view which has been
> > filled in 9.3 and should now be pg_upgraded into 9.4. If we don't have a
> > metapage yet and we want to add one we would need to rewrite the whole
> > 100GB which seems like a rather bad idea. Or how are you proposing to
> > add the metapage? You cannot add it to the end or somesuch.
>
> Oh, you are suggesting prepending a metapage to existing matviews
> (and tables?)?  So to check whether a view has been populated you
> not only look at the directory but open the file and read a page?
> Now I follow why you think this would be an issue.  I'm not sure I
> think that is the best solution, though.  In what way would it be
> better than adding info to pg_class or some other system table?

You can write/read it without having a database opened. Like in the
startup process.

Then you can abstract away the knowledge about that into
PageIsMetapage(relation, blockno) or such which then can be used by
vacuum, heapam et al in an extensible fashion.

This is far from a fully working solution though - you still have issues
with BEGIN;REFRESH ...; ROLLBACK; if you do it naively. Afair that was
what made Tom protest against this.

> Why would this be important for unlogged matviews but not unlogged
> tables?

Unlogged tables basically have some very raw version of this - the _init
fork. But yes, a more generic version would have been nice.

> >> I am not seeing this at all.  Given my comment above about how this
> >> works for pg_upgrade and pg_dump, can you explain how this is a
> >> problem?
> >
> > Well, that only works if there is a cheap way to add the new notation to
> > existing matview heaps which I don't see.
>
> We could perhaps reserve some space in the special area of the
> first page, if we can agree on a generous enough amount of space.

If we do it we should just use a whole page, no point in being to
cautious there imo.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services