Re: work on extending PostgreSQL to data integration systems

On Wed, Nov 01, 2006 at 02:13:47AM +0800, Lee Riquelmei wrote:
>
> Hello, everybody,
> I am developing a postgresql-based distributed relational data integration
> system which is like the IBM DB2 Information Integrator. And I have started
> a pgfoundry project: [1]http://pgfoundry.org/projects/ignite .(codes on it
> have already been obsolete.)
> The goal of the system is to provide a collection of virtual views in a
> pgsql database to integrate data from various distributed, heterogeneous and
> autonomous data sources.

You can do this now in a crude way using DBI-Link, which I am
developing now.
<http://pgfoundry.org/projects/dbi-link>

I'm very interested in your work, as DBI-Link's methods, all of which
act in user space, are somewhat crude. It has no hooks into the
planner, the optimizer, etc. It does have facilities which could use
such hooks if they were made available.

> To achieve this goal, I did my work in two aspects:
> (1) outside pgsql: The system utilizes various wrappers to interact with
> data sources. A well-defined wrapper interface is presented, which allows
> for implementations of concrete wrappers using any language , such as
> pl/java. In nature, a wrapper is a collection of pgsql functions which
> consists of the function for obtaining metadata of a data source, the
> function for exectuing SQL in data source and etc.
> (2) inside pgsql: Several modifictions to pgsql is used to improve the
> performance of processing data integration queries including the pipelined
> FunctionScan operator and support of query
> shipping(selection/projection/sort).
> The developing work is still ongoing, although an initial version of the
> system has already been used in a real application in China. This version
> consists of a wrapper for PostgreSQL using libpq and a wrapper for MS SQL
> SERVER using pl/java and JDBC. Currently, I have not published all sources
> and not provided good documents in the pgfoundry projects. But I will do it
> quickly. If anyone is interested in the system, pls contact me.
> I have a question. Does the PostgreSQL project have any plan of adding data
> integration functionalities in the PostgreSQL system as other DBMS vendors
> did?

The SQL:2003 standard includes a section called SQL/MED (Management of
External Data). I suspect that your implementation would be more
likely to go into the core if it were made compatible with this
standard.

Regards,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!