Skip site navigation (1) Skip section navigation (2)

Re: An idea for parallelizing COPY within one backend

From: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>
To: Brian Hurt <bhurt(at)janestcapital(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Postgresql-Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: An idea for parallelizing COPY within one backend
Date: 2008-02-27 15:56:14
Message-ID: 47C5881E.2040205@phlo.org (view raw or flat)
Thread:
Lists: pgsql-hackers
Brian Hurt wrote:
> Tom Lane wrote:
>> "Florian G. Pflug" <fgp(at)phlo(dot)org> writes:
>>> ...
>>> Neither the "dealer", nor the "workers" would need access to the either
>>> the shared memory or the disk, thereby not messing with the "one backend
>>> is one transaction is one session" dogma.
>>> ...
>> Unfortunately, this idea has far too narrow a view of what a datatype
>> input function might do.  Just for starters, consider "enum" input,
>> which certainly requires catalog access.  We have also explicitly
>> acknowledged the idea that datatype I/O functions might try to store
>> typmod-related data in some special catalog somewhere.

> Would it be possible to determine when the copy is starting that this 
> case holds, and not use the parallel parsing idea in those cases?

In theory, yes. In pratice, I don't want to be the one who has to answer 
to an angry user who just suffered a major drop in COPY performance 
after adding an ENUM column to his table.

I was thinking more along the line of letting a datatype specify a
function "void* ioprepare(typmod)" which returns some opaque object
specifying all that the input and output function needs to know.
We could than establish the rule that input/output functions may not 
access the catalog, and instead pass them a pointer to that opaque object.

All pretty pie-in-the-sky at the moment, though...

regards, Florian Pflug


In response to

Responses

pgsql-hackers by date

Next:From: Paul van den BogaardDate: 2008-02-27 15:58:57
Subject: Re: Proposed changes to DTrace probe implementation
Previous:From: Heikki LinnakangasDate: 2008-02-27 15:52:47
Subject: Re: An idea for parallelizing COPY within one backend

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group