Re: Parallel copy

From: Kyotaro Horiguchi <horikyota(dot)ntt(at)gmail(dot)com>
To: amit(dot)kapila16(at)gmail(dot)com
Cc: andrew(dot)dunstan(at)2ndquadrant(dot)com, minion(at)decodable(dot)me, thomas(dot)munro(at)gmail(dot)com, pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: Parallel copy
Date: 2020-02-18 10:59:22
Message-ID: 20200218.195922.1463471441610740886.horikyota.ntt@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

At Tue, 18 Feb 2020 15:59:36 +0530, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote in
> On Tue, Feb 18, 2020 at 7:28 AM Kyotaro Horiguchi
> <horikyota(dot)ntt(at)gmail(dot)com> wrote:
> >
> > In an extreme case, if we didn't see a QUOTE in a chunk, we cannot
> > know the chunk is in a quoted section or not, until all the past
> > chunks are parsed. After all we are forced to parse fully
> > sequentially as far as we allow QUOTE.
> >
>
> Right, I think the benefits of this as compared to single reader idea
> would be (a) we can save accessing shared memory for the most part of
> the chunk (b) for non-csv mode, even the tokenization (finding line
> boundaries) would also be parallel. OTOH, doing processing
> differently for csv and non-csv mode might not be good.

Agreed. So I think it's a good point of compromize.

> > On the other hand, if we allowed "COPY t FROM f WITH (FORMAT CSV,
> > QUOTE '')" in order to signal that there's no quoted section in the
> > file then all chunks would be fully concurrently parsable.
> >
>
> Yeah, if we can provide such an option, we can probably make parallel
> csv processing equivalent to non-csv. However, users might not like
> this as I think in some cases it won't be easier for them to tell
> whether the file has quoted fields or not. I am not very sure of this
> point.

I'm not sure how large portion of the usage contains quoted sections,
so I'm not sure how it is useful..

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Juan José Santamaría Flecha 2020-02-18 11:05:42 Re: Clean up some old cruft related to Windows
Previous Message Kyotaro Horiguchi 2020-02-18 10:50:16 Re: [HACKERS] WAL logging problem in 9.4.3?