Re: [PATCH] Simple progress reporting for COPY command

From: Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com>
To: Josef Šimánek <josef(dot)simanek(at)gmail(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: [PATCH] Simple progress reporting for COPY command
Date: 2021-01-07 15:54:51
Message-ID: 4c283c4b-bafe-2327-b41d-5caaa296dedf@enterprisedb.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On 1/7/21 12:06 PM, Josef Šimánek wrote:
> st 6. 1. 2021 v 22:44 odesílatel Tomas Vondra
> <tomas(dot)vondra(at)enterprisedb(dot)com> napsal:
>>
>> On 1/5/21 11:02 AM, Josef Šimánek wrote:
>>> I'm attaching the whole patch since commitfest failed to ingest the
>>> last incremental on CI.
>>>
>>
>> Yeah, the whole patch needs to be attached for the commitfest tester to
>> work correctly - it can't apply pieces from multiple messages, etc.
>>
>> Anyway, I pushed this last version of patch, after a couple more tweaks,
>> mainly to the docs - one place used pg_stat_copy_progress, the section
>> was not indexed properly, and so on.
>>
>> I see Matthias proposed to change "lines" to "tuples" - I only saw the
>> message after pushing, but I probably wouldn't make that change anyway.
>> The CSV docs seem to talk about lines, newlines etc. so it seems fine.
>> If not, we can change that.
>>
>> One more question, though - I now realize the lines_processed ignores
>> rows skipped because of BEFORE INSERT triggers. I wonder if that's the
>> right thing to do? Imagine you know the number of lines in a file. You
>> can't really use (lines_processed / total_lines) to measure progress,
>> because that may ignore many "skipped" rows. So maybe this should be
>> changed to count all rows. OTOH we still have bytes_processed.
>
> I think that should be fixed. It is called "lines_processed" not
> "lines_inserted". I'll take a look.
>

So we may either rename the column to "lines_inserted", or tweak the
code to count all processed lines. Or track both and have two columns.

regarss

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tomas Vondra 2021-01-07 15:56:26 Re: list of extended statistics on psql
Previous Message Fujii Masao 2021-01-07 15:51:45 Re: Add Information during standby recovery conflicts