Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

Hi Damir!

Your work looks like a very promising feature for production systems,
where data often needs to be loaded from external sources.

I've looked over the discussion and want to make a proposal -
when we load a bunch of records in database it does not make sense
to output errors to command output, and does not make sense to limit
error output to any number at all, because if we decided to load data
anyway - we would want to have a list (a file) with all records that were
discarded because of errors, with related error information, to, say,
deal with errors and process these records later. It looks like a reasonable
addition to your patch.

As a command output some limited number of error messages has much
less meaning than overall stats - records processed, records loaded,
records discarded, total number of errors.

For example you can look the Oracle SQL Loader feature, I hope this could
give some ideas for further improvements.

On Wed, Nov 2, 2022 at 11:46 AM Damir Belyalov <dam(dot)bel07(at)gmail(dot)com> wrote:

> Updated the patch:
> - Optimized and simplified logic of IGNORE_ERRORS
> - Changed variable names to more understandable ones
> - Added an analogue of MAX_BUFFERED_BYTES for safe buffer
>
>
> Regards,
> Damir Belyalov
> Postgres Professional
>
>>

--
Regards,
Nikita Malakhov
Postgres Professional
https://postgrespro.ru/