BUG #17089: "ERROR: invalid memory alloc request size ..." occurs when updating a fixed number of records

From: PG Bug reporting form <noreply(at)postgresql(dot)org>
To: pgsql-bugs(at)lists(dot)postgresql(dot)org
Cc: inconvenience(dot)notice(at)gmail(dot)com
Subject: BUG #17089: "ERROR: invalid memory alloc request size ..." occurs when updating a fixed number of records
Date: 2021-07-07 07:11:45
Message-ID: 17089-683fceecf0e385cf@postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

The following bug has been logged on the website:

Bug reference: 17089
Logged by: Yuetsuro Kobayashi
Email address: inconvenience(dot)notice(at)gmail(dot)com
PostgreSQL version: 13.3
Operating system: Windows 10 Pro and Windows 10 Pro for Workstations
Description:

[problem]
After importing a total of 357GB CSV files placed under 30 directories into
the table with COPY, UPDATE each record.
Import and UPDATE processing is performed by stored procedures.
After importing a CSV file that exceeds 150GB in total into the table,
"ERROR: invalid memory alloc request size 1677721600" always occurs in the
UPDATE of 77658539 records.
[environment]
CPU : Intel Core i9
OS : Windows10 Pro 64bit
DB : PostgreSQL13.3

shared_buffers = 16GB (25% of 64GB)

If you import the CSV file with the information of the 77658539th record
only and perform UPDATE processing, the processing will be completed
normally.
I divided 30 directories into 3 groups of 10 to verify ERROR.
Then, in order to keep the total CSV file of each group to 130GB or less,
the import & UPDATE process was executed in 3 steps.
As a result, the import & UPDATE process divided into three was successfully
processed. The import table has a bitserial type column.

From this result, I think that the phenomenon that causes the same "ERROR:
invalid memory alloc request size ..." problem all over the world has the
same problem in the latest PostgreSQL 13.3.

[question]
(1) Please provide information regarding the problem that "ERROR: invalid
memory alloc request size ..." occurs.
Why do I always happen "Error: Invalid memory allocation request size
1677721600" in UPDATE of 77658539th records after importing a CSV file
totaling more than 150GB into a table ?
Does EDB company solve this problem ?

(2) When "ERROR: invalid memory alloc request size ..." occurs, PostgreSQL
13.3 suppresses many log outputs in the executed stored procedure processing
and terminates.
Will this issue be fixed in the upcoming major release of PostgreSQL 14 ?

(3) If you uninstall PostgreSQL12 on a Windows OS machine and then install
PostgreSQL13, "Data directory initialization failure" and "parse error" will
occur.
This problem has been occurring frequently in also past major versions, and
I think it is still dragging the problem.
The bad news is that the only solution in that case is a silly solution
called an OS clean install.
I suspect this is due to the uninstaller installer, but has EDB already
resolved this issue ?

I manually deleted the data directory and the installation directory
(PostgreSQL \ 13) after the uninstall operation on Windows, but the
installation fails.
I also tried Registry Clean, but it still fails to install.

The problem is that some machines succeed or fail to reinstall after
uninstalling.
There are two machine information known at this time.
・ Windows10 Pro for Workstations → Reinstallation fails after
uninstalling
・ Windows10 Pro → After uninstalling, reinstallation succeeds

best regard.

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Etsuro Fujita 2021-07-07 07:20:45 Re: BUG #16631: postgres_fdw tries to insert into generated columns
Previous Message PG Bug reporting form 2021-07-07 06:33:07 BUG #17088: FailedAssertion in prepagg.c