Re: Extensions User Design

On Jun 23, 2009, at 10:44 AM, Dimitri Fontaine wrote:

> The contenders are extension, module, bundle and package. My vote is
> extension.

+1

> == v1.0 goals
>
> We're not trying to be feature complete on first round.
>
> * must have
>
> - dump & restore support (when upgrading a cluster or just restoring)
>
> - easy install and uninstall
>
> - support for home grown SQL/PLpgSQL only extensions in order to
> make life
> easier for in-house PG based development (you don't have to code
> in C to
> benefit from extensions)

It'd be nice if it supported other core languages like PL/Perl, but
it's okay if it doesn't on the first round (I'd likely want to use
some CPAN modules in a PL/Perl extension, anyway).

> - support for "basic" modules, providing a type and its operators and
> indexing support, such as ip4r, hstore, temporal, prefix and many
> others,
> you name it, of even simpler things like preprepare or
> backports/min_update.
>
> - support for procedural languages (a priori easily covered within
> basic
> modules but I'm not sure) like plproxy, pllolcode, pllua,
> plscheme, plsh
> et al.

Oh, here it is. So this goes with the point above, and can be
simplified to "support all procedural languages," yes?

> - support for all what you find in contrib/ for 8.4 (covered already?)
>
> * would be great (target later commit fest)
>
> - versioning support with upgrade in place facility (hooks?)

Yeah, we should standardize versioning somehow to make upgrading
easier. It should be a simple as possible, IMHO. If it tries to do too
much, you get stuck with great complexity.

> - supporting more than one version of the same module installed in
> the same
> time, possibly (I suppose always but...) in different schemas

Eh. This could be in 2.0 I think.

> - custom variables?

You mean GUC variables? That'd certainly be useful, but again,
probably not necessary for 1.0.

> - PostGIS complete support, with user data dependancy, even if an
> extensible typmod system would certainly solve this problem in a
> better
> place. Maybe someone will come up with another existing extension
> sharing
> the problem and not the typmod solution?

Well, PostGIS is itself an extension, no? What we need, then, is
dependency tracking.

> - a core team approved list of extensions (replacing contribs, maybe
> adding
> to it), where approved means code has been reviewed and the only
> reason
> why it's not in the core itself is that core team feels that it's
> not
> part of a RDBMS per-se, or feel like the code should be maintained
> and
> released separately until it gets some more field exposure... (think
> plproxy).

I hate the idea of "approved" extensions, but would love to see a kind
of "standard library" as a separate distribution that contains a bunch
of stuff that's commonly used. I'd want to steer clear of blessing by
the core team other than that, though, because then you start to get
into politics.

> * later please

Yah.

> - CPAN or ports like infrastructure for auto downloading a more or
> less
> prepared "bundle", place it at the right place on the filesystem and
> install it in the database(s) of choice

Yes, this would be nice. Also, integrated testing as with CPAN. I
happen to know of a really nice test framework we could use…

> - complex support for ad-hoc bootstrap of uncommon modules such as
> pljava

Not sure what this means; can you provide more detail?

> - dependancy graph solving and automatic installation, with depends,
> recommends and suggest sections and with rules/setup to choose
> what to
> pull in by default...

We'd likely have to store this information in some sort of system
table, too, yes?

> == dump & restore
>
> We want pg_dump to issue only one line per extension, the one
> installing the
> extension in the database, see syntax.
>
> == syntax
>
> Extensions will need metadata, and after reading several proposals,
> what I
> propose here is to have a first explicit step to register the
> extension name
> and metadata, then have "basic" tools to play with it.

Register with whom? I have to say that, although there is namespace
registration for CPAN, it's not required, and this is, in fact, a big
part of the reason for CPAN's success. There is no approval process
barrier to entry.

> === installing and removing an extension
>
> begin;
> install extension foo with search_path = foo;
> commit;

It would need something to ensure an appropriate version, too, no?

> Extensions authors are asked not to bother about search_path in
> their sql
> scripts so that it's easy for DBAs to decide where to install them.
> The with
> strange syntax is there to allow for the "install extension" command
> to
> default to, e.g., pg_extension, which won't typically be the first
> schema in
> the search_path.

And how will functions that call other functions within an extension
know that they're calling those functions in the appropriate schema? I
get this all the time with pgTAP: You can install it in its own
schema, but you have to include that schema in the search_path in
order for it to work, as some pgTAP functions call other pgTAP
functions with no schema-qualification.

> begin;
> drop extension foo [cascade];
> commit;
>
> The "cascade" option is there to care about reverse depends.

Would it fail if there were dependencies on the module in the
database, such as functions that use its functions, or tables that
depend on a custom data type?

> === creating extensions (authoring)
>
> The 'foo' extension author is meant to provide a +foo.sql+ file
> containing
> this:
>
> create extension foo
> with version 1.0
> install [script] 'foo.install.sql'
> uninstall [script] 'foo.uninstall.sql'
> upgrade function upgrade_foo(old version, new version)
> [ custom_variable_classes 'a,b'
> configuration file 'foo.conf' ]
> depends on bar version 0.3
> and on baz version >= 1.2;
>
> Here we suppose we have also a new datatype "version" to host the
> versionning information, with the associated operators. See
> http://packages.debian.org/sid/postgresql-8.3-debversion

I like this. Then the build file contains, essentially, just a SQL
command. That will make it easy for extension authors. However, they
might wish to include quite a lot of other metadata for the extension,
such as URLs for VC and bug tracking.

> Doing it this way, we skip the need to provide a way of telling "next
> comands are meant for creating SQL objects which belongs to such
> extension",
> at the expense of forcing authors to manage upgrades to add objects.

The install and uninstall script attributes should also allow either
full paths or, if just a simple file name, paths to the extensions
installation directory (currently $PGSQL/share/contrib).

> The upgrade function is mandatory, and has to return the installed
> version
> or null, meaning "please run the install script again, that's how I
> upgrade". The error management is to be made by means of RAISE
> EXCEPTION.

I'm not following you here. If I have a bunch of releases with a
number of changes to them, this function could get quite complex, I
should think. Also, in what language could it be written?

> If a specific function is to get called at install or uninstall
> time, it's
> easy enough to SELECT install_function(); from within the install
> script,
> after having defined it. To support this, internal GUCs (not exposed
> in
> postgresql.conf) will be provided and set by PG when running those
> scripts,
> named current_extension and current_extension_version.

Nice.

> == ACLs
>
> The "bulk" ACL management of an extension's objects is pushed to the
> globing
> support project for GRANT/REVOKE, so we don't have to speak about
> what it'll
> look like here :)
>
> == OS Filesystem Interaction
>
> PostgreSQL already provides standard paths where to install
> extensions by
> means of PGXS, and distribution packagers have been able to adapt
> those. We
> should just stick with this, meaning the problem is solved.

Sounds fine to me.

Best,

David