| From: | Simon Riggs <simon(at)2ndQuadrant(dot)com> |
|---|---|
| To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
| Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Dimitri Fontaine <dfontaine(at)hi-media(dot)com>, Emmanuel Cecchet <manu(at)asterdata(dot)com>, Emmanuel Cecchet <Emmanuel(dot)Cecchet(at)asterdata(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org> |
| Subject: | Re: COPY enhancements |
| Date: | 2009-10-12 13:13:15 |
| Message-ID: | 1255353195.15590.93.camel@ebony |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On Thu, 2009-10-08 at 11:01 -0400, Tom Lane wrote:
> So as far as I can see, the only form of COPY error handling that
> wouldn't be a cruel joke is to run a separate subtransaction for each
> row, and roll back the subtransaction on error. Of course the
> problems
> with that are (a) speed, (b) the 2^32 limit on command counter IDs
> would mean a max of 2^32 rows per COPY, which is uncomfortably small
> these days. Previous discussions of the problem have mentioned trying
> to batch multiple rows per subtransaction to alleviate both issues.
> Not easy of course, but that's why it's not been done yet. With a
> patch like this you'd also have (c) how to avoid rolling back the
> insertions into the logging table.
(d) using too many xids will force the system to begin immediate
wraparound-avoidance vacuuming to freeze rows.
Dimitri's pgloader is looking even more attractive, not least because it
exists and it works. (And is the reason I personally stopped considering
the COPY-error-logging feature as important).
--
Simon Riggs www.2ndQuadrant.com
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Teodor Sigaev | 2009-10-12 15:07:33 | Re: Re: [GENERAL] contrib/plantuner - enable PostgreSQL planner hints |
| Previous Message | Simon Riggs | 2009-10-12 13:06:33 | Hot Standby and 64+ subxids (was COPY enhancements) |